diff --git a/doc/book/admin/backups.rst b/doc/book/admin/backups.rst index 6e34d4c396..a2829db05f 100644 --- a/doc/book/admin/backups.rst +++ b/doc/book/admin/backups.rst @@ -37,15 +37,15 @@ that are made after the last snapshot are incremental backups. Therefore taking a backup is a matter of copying the snapshot and WAL files. 1. Use ``tar`` to make a (possibly compressed) copy of the latest .snap and .xlog - files on the :ref:`memtx_dir ` and - :ref:`wal_dir ` directories. + files on the :ref:`snapshot.dir ` and + :ref:`wal.dir ` directories. 2. If there is a security policy, encrypt the .tar file. 3. Copy the .tar file to a safe place. Later, restoring the database is a matter of taking the .tar file and putting -its contents back in the ``memtx_dir`` and ``wal_dir`` directories. +its contents back in the ``snapshot.dir`` and ``wal.dir`` directories. .. _admin-backups-hot_backup_vinyl_memtx: diff --git a/doc/code_snippets/snippets/config/instances.enabled/persistence_snapshot/config.yaml b/doc/code_snippets/snippets/config/instances.enabled/persistence_snapshot/config.yaml new file mode 100644 index 0000000000..ac71476864 --- /dev/null +++ b/doc/code_snippets/snippets/config/instances.enabled/persistence_snapshot/config.yaml @@ -0,0 +1,15 @@ +groups: + group001: + replicasets: + replicaset001: + instances: + instance001: + snapshot: + dir: 'var/lib/{{ instance_name }}/snapshots' + count: 3 + by: + interval: 7200 + wal_size: 1000000000000000000 + iproto: + listen: + - uri: '127.0.0.1:3301' \ No newline at end of file diff --git a/doc/code_snippets/snippets/config/instances.enabled/persistence_snapshot/instances.yml b/doc/code_snippets/snippets/config/instances.enabled/persistence_snapshot/instances.yml new file mode 100644 index 0000000000..9d7cb5e7c2 --- /dev/null +++ b/doc/code_snippets/snippets/config/instances.enabled/persistence_snapshot/instances.yml @@ -0,0 +1 @@ +instance001: \ No newline at end of file diff --git a/doc/code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml b/doc/code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml new file mode 100644 index 0000000000..a073a3f2e2 --- /dev/null +++ b/doc/code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml @@ -0,0 +1,21 @@ +groups: + group001: + replicasets: + replicaset001: + instances: + instance001: + wal: + dir: 'var/lib/{{ instance_name }}/wals' + mode: 'write' + dir_rescan_delay: 3 + cleanup_delay: 18000 + max_size: 268435456 + ext: + new: true + old: true + spaces: + bands: + old: false + iproto: + listen: + - uri: '127.0.0.1:3301' \ No newline at end of file diff --git a/doc/code_snippets/snippets/config/instances.enabled/persistence_wal/instances.yml b/doc/code_snippets/snippets/config/instances.enabled/persistence_wal/instances.yml new file mode 100644 index 0000000000..9d7cb5e7c2 --- /dev/null +++ b/doc/code_snippets/snippets/config/instances.enabled/persistence_wal/instances.yml @@ -0,0 +1 @@ +instance001: \ No newline at end of file diff --git a/doc/concepts/atomic/thread_model.rst b/doc/concepts/atomic/thread_model.rst index fb3e90d4e9..28f5806850 100644 --- a/doc/concepts/atomic/thread_model.rst +++ b/doc/concepts/atomic/thread_model.rst @@ -69,7 +69,7 @@ There are also several supplementary threads that serve additional capabilities: Separate threads are required because each replica can point to a different position in the log and can run at different speeds. * There is a thread pool for ad hoc asynchronous tasks, - such as a DNS resolver or :ref:`fsync `. + such as a DNS resolver or :ref:`fsync `. * There are OpenMP threads used to parallelize sorting (hence, to parallelize building :ref:`indexes `). diff --git a/doc/concepts/configuration.rst b/doc/concepts/configuration.rst index d1f250bc9f..e709c66ec2 100644 --- a/doc/concepts/configuration.rst +++ b/doc/concepts/configuration.rst @@ -426,7 +426,8 @@ When the limit is reached, ``INSERT`` or ``UPDATE`` requests fail with :ref:`ER_ Snapshots and write-ahead logs ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The ``snapshot.dir`` and ``wal.dir`` options can be used to configure directories for storing snapshots and write-ahead logs. +The :ref:`snapshot.dir ` and :ref:`wal.dir ` +options can be used to configure directories for storing snapshots and write-ahead logs. For example, you can place snapshots and write-ahead logs on different hard drives for better reliability. .. code-block:: yaml @@ -438,6 +439,7 @@ For example, you can place snapshots and write-ahead logs on different hard driv dir: '/media/drive2/wals' To learn more about the persistence mechanism in Tarantool, see the :ref:`Persistence ` section. +Read more about snapshot and WAL configuration: :ref:`Persistence `. @@ -447,6 +449,7 @@ To learn more about the persistence mechanism in Tarantool, see the :ref:`Persis configuration/configuration_etcd configuration/configuration_code + configuration/configuration_persistence configuration/configuration_connections configuration/configuration_credentials configuration/configuration_authentication diff --git a/doc/concepts/configuration/configuration_persistence.rst b/doc/concepts/configuration/configuration_persistence.rst new file mode 100644 index 0000000000..07a080aa72 --- /dev/null +++ b/doc/concepts/configuration/configuration_persistence.rst @@ -0,0 +1,284 @@ +.. _configuration_persistence: + +Persistence +=========== + +To ensure data persistence, Tarantool provides the abilities to: + +* Record each data change request into a :ref:`write-ahead log ` (WAL) file (``.xlog`` files). +* Take :ref:`snapshots ` that contain an on-disk copy of the entire data set for a given moment + (``.snap`` files). It is possible to set automatic snapshot creation using the :ref:`checkpoint daemon `. + +During the recovery process, Tarantool can load the latest snapshot file and then read the requests from the WAL files, +produced after this snapshot was made. +This topic describes how to configure: + +* the snapshot creation in the :ref:`snapshot ` section of a :ref:`YAML configuration `. +* the recording to the write-ahead log in the :ref:`wal ` section of a YAML configuration. + +To learn more about the persistence mechanism in Tarantool, see the :ref:`Persistence ` section. +The formats of WAL and snapshot files are described in detail in the :ref:`File formats ` section. + +.. _configuration_persistence_snapshot: + +Configure the snapshots +----------------------- + +**Example on GitHub**: `snapshot `_ + +This section describes how to define snapshot settings in the :ref:`snapshot ` section of a YAML configuration. + +.. _configuration_persistence_snapshot_creation: + +Set up automatic snapshot creation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In Tarantool, it is possible to automate the :ref:`snapshot creation `. +Automatic creation is enabled by default and can be configured in two ways: + +* A new snapshot is taken once in a given period (see :ref:`snapshot.by.interval `). +* A new snapshot is taken once the size of all WAL files created since the last snapshot exceeds a given limit + (see :ref:`snapshot.by.wal_size `). + +The ``snapshot.by.interval`` option sets up the :ref:`checkpoint daemon ` +that takes a new snapshot every ``snapshot.by.interval`` seconds. +If the ``snapshot.by.interval`` option is set to zero, the checkpoint daemon is disabled. + +The ``snapshot.by.wal_size`` option defines the maximum size in bytes for all WAL files created since the last snapshot taken. +Once this size is exceeded, the checkpoint daemon takes a snapshot. Then, :ref:`Tarantool garbage collector ` +deletes the old WAL files. + +The example shows how to specify the ``snapshot.by.interval`` and the ``snapshot.by.wal_size`` options: + +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_snapshot/config.yaml + :language: yaml + :start-at: by: + :end-at: 1000000000000000000 + :dedent: + +In the example, a new snapshot is created in two cases: + +* every 2 hours (every 7200 seconds) +* when the size for all WAL files created since the last snapshot reaches the size of 1e18 (1000000000000000000) bytes. + +.. _configuration_persistence_snapshot_dir: + +Specify a directory for snapshot files +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To configure a directory where the snapshot files are stored, use the :ref:`snapshot.dir ` +configuration option. +The example below shows how to specify a snapshot directory for ``instance001`` explicitly: + +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_snapshot/config.yaml + :language: yaml + :start-at: instance001: + :end-at: 'var/lib/{{ instance_name }}/snapshots' + :dedent: + +By default, WAL files and snapshot files are stored in the same directory ``var/lib/{{ instance_name }}``. +However, you can specify different directories for them. +For example, you can place snapshots and write-ahead logs on different hard drives for better reliability: + +.. code-block:: yaml + + instance001: + snapshot: + dir: '/media/drive1/snapshots' + wal: + dir: '/media/drive2/wals' + +.. _configuration_persistence_snapshot_count: + +Configure a maximum number of stored snapshots +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can set a limit on the number of snapshots stored in the :ref:`snapshot.dir ` +directory using the :ref:`snapshot.count ` option. +Once the number of snapshots reaches the given limit, :ref:`Tarantool garbage collector ` +deletes the oldest snapshot file and any associated WAL files after the new snapshot is taken. + +In the example below, the snapshot is created every two hours (every 7200 seconds) until there are three snapshots in the +``snapshot.dir`` directory. +After creating a new snapshot (the fourth one), the oldest snapshot and the corresponding WALs are deleted. + +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_snapshot/config.yaml + :language: yaml + :start-at: count: + :end-at: 7200 + :dedent: + +.. _configuration_persistence_wal: + +Configure the write-ahead log +----------------------------- + +**Example on GitHub**: `wal `_ + +This section describes how to define WAL settings in the :ref:`wal ` section of a YAML configuration. + +.. _configuration_persistence_wal_mode: + +Set the WAL mode +~~~~~~~~~~~~~~~~ + +The recording to the write-ahead log is enabled by default. +It means that if an instance restart occurs, the data will be recovered. +The recording to the WAL can be configured using the :ref:`wal.mode ` configuration option. + +There are two modes that enable writing to the WAL: + +* ``write`` (default) -- enable WAL and write the data without waiting for the data to be flushed to the storage device. +* ``fsync`` -- enable WAL and ensure that the record is written to the storage device. + +The example below shows how to specify the ``write`` WAL mode: + +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml + :language: yaml + :start-at: mode: + :end-at: 'write' + :dedent: + +To turn the WAL writer off, set the ``wal.mode`` option to ``none``. + +.. _configuration_persistence_wal_dir: + +Specify a directory for WAL files +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To configure a directory where the WAL files are stored, use the :ref:`wal.dir ` configuration option. +The example below shows how to specify a directory for ``instance001`` explicitly: + +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml + :language: yaml + :start-at: instance001: + :end-at: 'var/lib/{{ instance_name }}/wals' + :dedent: + + +.. _configuration_persistence_wal_rescan: + +Set an interval between scans +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In case of :ref:`replication ` or :ref:`hot standby ` mode, +Tarantool scans for changes in the WAL files every :ref:`wal.dir_rescan_delay ` +seconds. The example below shows how to specify the interval between scans: + +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml + :language: yaml + :start-at: dir_rescan_delay + :end-before: cleanup_delay + :dedent: + +.. _configuration_persistence_wal_maxsize: + +Set a maximum size for the WAL file +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A new WAL file is created when the current one reaches the :ref:`wal.max_size ` +size. The configuration for this option might look as follows: + +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml + :language: yaml + :start-at: max_size + :end-at: 268435456 + :dedent: + +.. _configuration_persistence_wal_rescan: + +Set a delay for the garbage collector +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In Tarantool, the :ref:`checkpoint daemon ` +takes new snapshots at the given interval (see :ref:`snapshot.by.interval `). +After an instance restart, the Tarantool garbage collector deletes the old WAL files. + +To delay the immediate deletion of WAL files, use the :ref:`wal.cleanup_delay ` +configuration option. The delay eliminates possible erroneous situations when the master deletes WALs +needed by :ref:`replicas ` after restart. +As a consequence, replicas sync with the master faster after its restart and +don't need to download all the data again. + +In the example, the delay is set to 5 hours (18000 seconds): + +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml + :language: yaml + :start-at: cleanup_delay + :end-at: 18000 + :dedent: + +.. _configuration_persistence_wal_ext: + +Specify the WAL extensions +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In Tarantool Enterprise, you can store an old and new tuple for each CRUD operation performed. +A detailed description and examples of the WAL extensions are provided in the :ref:`WAL extensions ` section. + +See also: :ref:`wal.ext.* ` configuration options. + +.. _configuration_persistence_checkpoint_daemon: + +Checkpoint daemon +----------------- + +The checkpoint daemon (snapshot daemon) is a constantly running :ref:`fiber `. +The checkpoint daemon creates a schedule for the periodic snapshot creation based on +the :ref:`configuration options ` and the speed of file size growth. +If enabled, the daemon makes new :ref:`snapshot ` (``.snap``) files according to this schedule. + +The work of the checkpoint daemon is based on the following configuration options: + +* :ref:`snapshot.by.interval ` -- a new snapshot is taken once in a given period. +* :ref:`snapshot.by.wal_size ` -- a new snapshot is taken once the size + of all WAL files created since the last snapshot exceeds a given limit. + +If necessary, the checkpoint daemon also activates the :ref:`Tarantool garbage collector ` +that deletes old snapshots and WAL files. + +.. _configuration_persistence_garbage_collector: + +Tarantool garbage collector +--------------------------- + +Tarantool garbage collector can be activated by the :ref:`checkpoint daemon `. +The garbage collector tracks the snapshots that are to be :ref:`relayed to a replica ` or needed +by other consumers. When the files are no longer needed, Tarantool garbage collector deletes them. + +.. NOTE:: + + The garbage collector called by the checkpoint daemon is distinct from the `Lua garbage collector `_ + which is for Lua objects, and distinct from the Tarantool garbage collector that specializes in :ref:`handling shard buckets `. + +This garbage collector is called as follows: + +* When the number of snapshots reaches the limit of :ref:`snapshot.count ` size. + After a new snapshot is taken, Tarantool garbage collector deletes the oldest snapshot file and any associated WAL files. + +* When the size of all WAL files created since the last snapshot reaches the limit of :ref:`snapshot.by.wal_size `. + Once this size is exceeded, the checkpoint daemon takes a snapshot, then the garbage collector deletes the old WAL files. + +If an old snapshot file is deleted, the Tarantool garbage collector also deletes +any :ref:`write-ahead log (.xlog) ` files that meet the following conditions: + +* The WAL files are older than the snapshot file. +* The WAL files contain information present in the snapshot file. + +Tarantool garbage collector also deletes obsolete vinyl ``.run`` files. + +Tarantool garbage collector doesn't delete a file in the following cases: + +* A backup is running, and the file has not been backed up + (see :ref:`Hot backup `). + +* Replication is running, and the file has not been relayed to a replica + (see :ref:`Replication architecture `), + +* A replica is connecting. + +* A replica has fallen behind. + The progress of each replica is tracked; if a replica's position is far + from being up to date, then the server stops to give it a chance to catch up. + If an administrator concludes that a replica is permanently down, then the + correct procedure is to restart the server, or (preferably) :ref:`remove the replica from the cluster `. diff --git a/doc/concepts/data_model/operations.rst b/doc/concepts/data_model/operations.rst index 602c202539..30a8243066 100644 --- a/doc/concepts/data_model/operations.rst +++ b/doc/concepts/data_model/operations.rst @@ -150,7 +150,7 @@ resource usage of each function. important than the others. * - WAL settings - The important setting for the write-ahead log is - :ref:`wal_mode `. + :ref:`wal.mode `. If the setting causes no writing or delayed writing, this factor is unimportant. If the setting causes every data-change request to wait diff --git a/doc/concepts/data_model/persistence.rst b/doc/concepts/data_model/persistence.rst index 0545fdbfef..dce7443f67 100644 --- a/doc/concepts/data_model/persistence.rst +++ b/doc/concepts/data_model/persistence.rst @@ -12,7 +12,7 @@ In such case, Tarantool restores the data from WAL files by reading them and redoing the requests. This is called the "recovery process". You can change the timing of the WAL writer or turn it off by setting -the :ref:`wal_mode `. +the :ref:`wal.mode `. Tarantool also maintains a set of :ref:`snapshot files `. These files contain an on-disk copy of the entire data set for a given moment. @@ -24,8 +24,8 @@ After creating a new snapshot, the earlier WAL files can be removed to free up s To force immediate creation of a snapshot file, use the :doc:`box.snapshot() ` function. To enable the automatic creation of snapshot files, use Tarantool's -:ref:`checkpoint daemon `. -The checkpoint daemon sets intervals for forced checkpoints. It makes sure that the states +:ref:`checkpoint daemon `. +The checkpoint daemon sets intervals for forced snapshots. It makes sure that the states of both memtx and vinyl storage engines are synchronized and saved to disk, and automatically removes earlier WAL files. @@ -33,10 +33,11 @@ Snapshot files can be created even if there is no WAL file. .. NOTE:: - The memtx engine makes only regular checkpoints with the interval set in - :ref:`checkpoint daemon ` configuration. + The memtx engine takes only regular snapshots with the interval set in + the checkpoint daemon configuration. The vinyl engine runs checkpointing in the background at all times. See the :ref:`Internals ` section for more details about the WAL writer and the recovery process. +To learn more about the configuration of the checkpoint daemon and WAL, check the :ref:`Persistence ` page. \ No newline at end of file diff --git a/doc/dev_guide/internals/file_formats.rst b/doc/dev_guide/internals/file_formats.rst index 72ef740f47..edf7b55c42 100644 --- a/doc/dev_guide/internals/file_formats.rst +++ b/doc/dev_guide/internals/file_formats.rst @@ -1,84 +1,86 @@ -.. _internals-data_persistence: +.. _internals-data_persistence: File formats ============ .. _internals-wal: -Data persistence and the WAL file format ----------------------------------------- +The WAL file format +------------------- -To maintain data persistence, Tarantool writes each data change request (insert, -update, delete, replace, upsert) into a write-ahead log (WAL) file in the -:ref:`wal_dir ` directory. A new WAL file is created -when the current one reaches the :ref:`wal_max_size ` size. -Each data change request gets assigned a continuously growing 64-bit log sequence +To maintain :ref:`data persistence `, Tarantool writes each data change request (insert, +update, delete, replace, upsert) to a write-ahead log (WAL) file in the +:ref:`wal.dir ` directory. +Each data change request is assigned a continuously growing 64-bit log sequence number. The name of the WAL file is based on the log sequence number of the first record in the file, plus an extension ``.xlog``. +A new WAL file is created +when the current one reaches the :ref:`wal_max_size ` size. + +Each WAL record contains: + +* a log sequence number +* a data change request (formatted as in :ref:`Tarantool's binary protocol `) + +* a header +* some metadata +* the data formatted according to `msgpack `_ rules. + +To see the hexadecimal bytes of the given WAL file, use the ``hexdump`` command: + +.. code-block:: console -Apart from a log sequence number and the data change request (formatted as in -:ref:`Tarantool's binary protocol `), -each WAL record contains a header, some metadata, and then the data formatted -according to `msgpack `_ rules. -For example, this is what the WAL file looks like after the first INSERT request -("s:insert({1})") for the sandbox database created in our -:ref:`"Getting started" exercises `. -On the left are the hexadecimal bytes that you would see with: - -.. code-block:: console - - $ hexdump 00000000000000000000.xlog - -and on the right are comments. - -.. code-block:: none - - Hex dump of WAL file Comment - -------------------- ------- - 58 4c 4f 47 0a "XLOG\n" - 30 2e 31 33 0a "0.13\n" = version - 53 65 72 76 65 72 3a 20 "Server: " - 38 62 66 32 32 33 65 30 2d [Server UUID]\n - 36 39 31 34 2d 34 62 35 35 - 2d 39 34 64 32 2d 64 32 62 - 36 64 30 39 62 30 31 39 36 - 0a - 56 43 6c 6f 63 6b 3a 20 "Vclock: " - 7b 7d "{}" = vclock value, initially blank - ... (not shown = tuples for system spaces) - d5 ba 0b ab Magic row marker always = 0xab0bbad5 - 19 Length, not including length of header, = 25 bytes - 00 Record header: previous crc32 - ce 8c 3e d6 70 Record header: current crc32 - a7 cc 73 7f 00 00 66 39 Record header: padding - 84 msgpack code meaning "Map of 4 elements" follows - 00 02 element#1: tag=request type, value=0x02=IPROTO_INSERT - 02 01 element#2: tag=server id, value=0x01 - 03 04 element#3: tag=lsn, value=0x04 - 04 cb 41 d4 e2 2f 62 fd d5 d4 element#4: tag=timestamp, value=an 8-byte "Float64" - 82 msgpack code meaning "map of 2 elements" follows - 10 cd 02 00 element#1: tag=space id, value=512, big byte first - 21 91 01 element#2: tag=tuple, value=1-element fixed array={1} + $ hexdump 00000000000000000000.xlog + +For example, the WAL file after the first INSERT request might look the following way: + +.. code-block:: none + + Hex dump of WAL file Comment + -------------------- ------- + 58 4c 4f 47 0a "XLOG\n" + 30 2e 31 33 0a "0.13\n" = version + 53 65 72 76 65 72 3a 20 "Server: " + 38 62 66 32 32 33 65 30 2d [Server UUID]\n + 36 39 31 34 2d 34 62 35 35 + 2d 39 34 64 32 2d 64 32 62 + 36 64 30 39 62 30 31 39 36 + 0a + 56 43 6c 6f 63 6b 3a 20 "Vclock: " + 7b 7d "{}" = vclock value, initially blank + ... (not shown = tuples for system spaces) + d5 ba 0b ab Magic row marker always = 0xab0bbad5 + 19 Length, not including length of header, = 25 bytes + 00 Record header: previous crc32 + ce 8c 3e d6 70 Record header: current crc32 + a7 cc 73 7f 00 00 66 39 Record header: padding + 84 msgpack code meaning "Map of 4 elements" follows + 00 02 element#1: tag=request type, value=0x02=IPROTO_INSERT + 02 01 element#2: tag=server id, value=0x01 + 03 04 element#3: tag=lsn, value=0x04 + 04 cb 41 d4 e2 2f 62 fd d5 d4 element#4: tag=timestamp, value=an 8-byte "Float64" + 82 msgpack code meaning "map of 2 elements" follows + 10 cd 02 00 element#1: tag=space id, value=512, big byte first + 21 91 01 element#2: tag=tuple, value=1-element fixed array={1} Tarantool processes requests atomically: a change is either accepted and recorded -in the WAL, or discarded completely. Let's clarify how this happens, using the -REPLACE request as an example: +in the WAL, or discarded completely. To clarify how this happens, see the example with the REPLACE request below: -1. The server instance attempts to locate the original tuple by primary key. If found, a +#. The server instance attempts to locate the original tuple by primary key. If found, a reference to the tuple is retained for later use. -2. The new tuple is validated. If for example it does not contain an indexed +#. The new tuple is validated. If for example it does not contain an indexed field, or it has an indexed field whose type does not match the type according to the index definition, the change is aborted. -3. The new tuple replaces the old tuple in all existing indexes. +#. The new tuple replaces the old tuple in all existing indexes. -4. A message is sent to the WAL writer running in a separate thread, requesting that +#. A message is sent to the WAL writer running in a separate thread, requesting that the change be recorded in the WAL. The instance switches to work on the next request until the write is acknowledged. -5. On success, a confirmation is sent to the client. On failure, a rollback - procedure is begun. During the rollback procedure, the transaction processor +#. On success, a confirmation is sent to the client. On failure, a rollback + procedure begins. During the rollback procedure, the transaction processor rolls back all changes to the database which occurred after the first failed change, from latest to oldest, up to the first failed change. All rolled back requests are aborted with :errcode:`ER_WAL_IO ` error. No new @@ -91,49 +93,50 @@ database performance doesn't degrade even if all requests refer to the same key in the same space. The transaction processor thread communicates with the WAL writer thread using -asynchronous (yet reliable) messaging; the transaction processor thread, not -being blocked on WAL tasks, continues to handle requests quickly even at high +asynchronous (yet reliable) messaging. +The transaction processor thread, not being blocked on WAL tasks, continues to handle requests quickly even at high volumes of disk I/O. A response to a request is sent as soon as it is ready, even if there were earlier incomplete requests on the same connection. In particular, SELECT performance, even for SELECTs running on a connection packed with UPDATEs and DELETEs, remains unaffected by disk load. The WAL writer employs a number of durability modes, as defined in configuration -variable :ref:`wal_mode `. It is possible to turn the write-ahead -log completely off, by setting -:ref:`wal_mode ` to *none*. Even -without the write-ahead log it's still possible to take a persistent copy of the +variable :ref:`wal.mode `. +It is possible to turn the write-ahead log completely off, by setting the ``wal_mode`` option to *none*. +Even without the write-ahead log it's still possible to take a persistent copy of the entire data set with the :ref:`box.snapshot() ` request. -An .xlog file always contains changes based on the primary key. +An ``.xlog`` file always contains changes based on the primary key. Even if the client requested an update or delete using -a secondary key, the record in the .xlog file will contain the primary key. +a secondary key, the record in the .xlog file contains the primary key. -.. _internals-snapshot: +.. _internals-snapshot: The snapshot file format ------------------------ -The format of a snapshot .snap file is nearly the same as the format of a WAL .xlog file. -However, the snapshot header differs: it contains the instance's global unique identifier -and the snapshot file's position in history, relative to earlier snapshot files. -Also, the content differs: an .xlog file may contain records for any data-change -requests (inserts, updates, upserts, and deletes), a .snap file may only contain records -of inserts to memtx spaces. +The format of a snapshot (``.snap``) file is the following: + +* The snapshot header contains the instance's global unique identifier + and the snapshot file's position in history, relative to earlier snapshot files. + +* The snapshot content contains the records of inserts to memtx spaces. + That differs from the content of an ``.xlog`` file that may contain records for any data-change requests + (inserts, updates, upserts, and deletes). + +Primarily, the records in the snapshot file have the following order: -Primarily, the .snap file's records are ordered by space id. Therefore the records of -system spaces -- such as ``_schema``, ``_space``, ``_index``, ``_func``, ``_priv`` -and ``_cluster`` -- will be at the start of the .snap file, before the records of -any spaces that were created by users. +* System spaces (id >= 256 && id <= 511), ordered by ID. +* Non-system spaces, ordered by ID. -Secondarily, the .snap file's records are ordered by primary key within space id. +Secondarily, the .snap file's records are ordered by primary key within space ID. .. _box_protocol-xlog: Example ------- -The header of a ``.snap`` or ``.xlog`` file looks like: +The header of a ``.snap`` or ``.xlog`` file might look in the following way: .. code-block:: none diff --git a/doc/enterprise/audit.rst b/doc/enterprise/audit.rst index 927d5f7266..c2ea40409e 100644 --- a/doc/enterprise/audit.rst +++ b/doc/enterprise/audit.rst @@ -144,21 +144,17 @@ Tarantool records all incoming data in the write-ahead log (WAL). The WAL must be enabled to ensure that data will be recovered in case of a possible instance restart. -Secure values of ``wal_mode`` are ``write`` and ``fsync``: +Secure values of the :ref:`wal.mode ` configuration option are ``write`` and ``fsync``: -.. code-block:: tarantoolsession - - tarantool> box.cfg.wal_mode - --- - - write +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml + :language: yaml + :start-at: wal: + :end-at: 'write' + :dedent: An exclusion from this requirement is when the instance is processing data, -which can be freely rejected. -For example, when Tarantool is used for caching. -Then WAL can be disabled to reduce i/o load. - -For more details, see the -:ref:`wal_mode reference `. +which can be freely rejected - for example, when Tarantool is used for caching. +In this case, WAL can be disabled to reduce i/o load. The logging level is INFO or higher ----------------------------------- diff --git a/doc/enterprise/flight_recorder.rst b/doc/enterprise/flight_recorder.rst index 36061b21bd..67184849b7 100644 --- a/doc/enterprise/flight_recorder.rst +++ b/doc/enterprise/flight_recorder.rst @@ -17,8 +17,8 @@ to :ref:`crashing ` a Tarantool instance. .. _enterprise-flight-recorder_enable: -Enabling the flight recorder ----------------------------- +Enable the flight recorder +-------------------------- The flight recorder is disabled by default and can be enabled and configured for a specific Tarantool instance. @@ -33,7 +33,7 @@ configuration option to ``true``. After ``flightrec.enabled`` is set to ``true``, the flight recorder starts collecting data in the flight recording file ``current.ttfr``. -This file is stored in the ``snapshot.dir`` directory. +This file is stored in the :ref:`snapshot.dir ` directory. By default, the directory is ``var/lib/{{ instance_name }}/.ttfr``. If the instance crashes and reboots, Tarantool rotates the flight recording: @@ -49,8 +49,8 @@ Tarantool continues writing to the existing ``current.ttfr`` file after restart. .. _enterprise-flight-recorder_configure: -Configuring a flight recorder ------------------------------ +Configure a flight recorder +--------------------------- When the flight recorder is enabled, you can set the options related to :ref:`logging `, :ref:`metrics `, and storing the :ref:`request and response ` data. diff --git a/doc/enterprise/wal_extensions.rst b/doc/enterprise/wal_extensions.rst index c07cc5d0fb..5be4967530 100644 --- a/doc/enterprise/wal_extensions.rst +++ b/doc/enterprise/wal_extensions.rst @@ -1,4 +1,4 @@ -.. _wal_extensions: +.. _wal_extensions: WAL extensions ============== @@ -8,48 +8,51 @@ For example, you can enable storing an old and new tuple for each CRUD operation This information might be helpful for implementing a CDC (Change Data Capture) utility that transforms a data replication stream. +See also: :ref:`Configure the write-ahead log `. -.. _wal_extensions_configuration: +.. _wal_extensions_configuration: Configuration ------------- -To configure WAL extensions, use the ``wal_ext`` :ref:`configuration property `. -Inside the ``wal_ext`` block, you can enable storing old and new tuples as follows: +WAL extensions are disabled by default. +To configure them, use the :ref:`wal.ext.* ` configuration options. +Inside the ``wal.ext`` block, you can enable storing old and new tuples as follows: -* Set the ``old`` and ``new`` options to ``true`` to store old and new tuples in a write-ahead log for all spaces. +* To store old and new tuples in a write-ahead log for all spaces, set the + :ref:`wal.ext.old ` and :ref:`wal.ext.new ` + options to ``true``: - .. code-block:: lua + .. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence/config.yaml + :language: yaml + :start-at: wal: + :end-at: old: true + :dedent: - box.cfg { - wal_ext = { old = true, new = true } - } +* To adjust these options for specific spaces, specify the :ref:`wal.ext.spaces ` option: -* To adjust these options for specific spaces, use the ``spaces`` option. + .. code-block:: yaml - .. code-block:: lua + wal: + ext: + old: true + new: true + spaces: + space1: + old: false + space2: + new: false - box.cfg { - wal_ext = { - old = true, new = true, - spaces = { - space1 = { old = false }, - space2 = { new = false } - } - } - } - - - The configuration for specific spaces has priority over the global configuration, - so only new tuples are added to the log for ``space1`` and only old tuples for ``space2``. + The configuration for specific spaces has priority over the configuration in the ``wal.ext.new`` and ``wal.ext.old`` + options. + It means that only new tuples are added to the log for ``space1`` and only old tuples for ``space2``. Note that records with additional fields are :ref:`replicated ` as follows: * If a replica doesn't support the extended format configured on a master, auxiliary fields are skipped. -* If a replica and master have different configurations for WAL records, a master's configuration is ignored. - +* If a replica and master have different configurations for WAL records, the master's configuration is ignored. -.. _wal_extensions_example: +.. _wal_extensions_example: Example ------- diff --git a/doc/reference/configuration/cfg_basic.rst b/doc/reference/configuration/cfg_basic.rst index 495b7f1eb2..aee6609527 100644 --- a/doc/reference/configuration/cfg_basic.rst +++ b/doc/reference/configuration/cfg_basic.rst @@ -100,9 +100,12 @@ Since version 1.7.4. - A directory where memtx stores snapshot (.snap) files. Can be relative to - :ref:`work_dir `. If not specified, defaults to - ``work_dir``. See also :ref:`wal_dir `. + A directory where memtx stores snapshot (``.snap``) files. + A relative path in this option is interpreted as relative to :ref:`work_dir `. + + By default, snapshots and WAL files are stored in the same directory. + However, you can set different values for the ``memtx_dir`` and :ref:`wal_dir ` options + to store them on different physical disks for performance matters. | Type: string | Default: "." @@ -230,11 +233,12 @@ Since version 1.6.2. - A directory where write-ahead log (.xlog) files are stored. Can be relative - to :ref:`work_dir `. Sometimes ``wal_dir`` and - :ref:`memtx_dir ` are specified with different values, so - that write-ahead log files and snapshot files can be stored on different - disks. If not specified, defaults to ``work_dir``. + A directory where write-ahead log (``.xlog``) files are stored. + A relative path in this option is interpreted as relative to :ref:`work_dir `. + + By default, WAL files and snapshots are stored in the same directory. + However, you can set different values for the ``wal_dir`` and :ref:`memtx_dir ` options + to store them on different physical disks for performance matters. | Type: string | Default: "." diff --git a/doc/reference/configuration/cfg_binary_logging_snapshots.rst b/doc/reference/configuration/cfg_binary_logging_snapshots.rst index 83f29695b6..a649e8663d 100644 --- a/doc/reference/configuration/cfg_binary_logging_snapshots.rst +++ b/doc/reference/configuration/cfg_binary_logging_snapshots.rst @@ -36,8 +36,8 @@ Since version 1.7.4. The maximum number of bytes in a single write-ahead log file. - When a request would cause an .xlog file to become larger than - ``wal_max_size``, Tarantool creates another WAL file. + When a request would cause an ``.xlog`` file to become larger than + ``wal_max_size``, Tarantool creates a new WAL file. | Type: integer | Default: 268435456 (256 * 1024 * 1024) bytes @@ -58,7 +58,7 @@ locations and moving snapshots to a separate disk. The limit also affects what :ref:`box.stat.vinyl().regulator ` - may show for the write rate of dumps to .run and .index files. + may show for the write rate of dumps to ``.run`` and ``.index`` files. | Type: float | Default: null @@ -73,12 +73,14 @@ Specify fiber-WAL-disk synchronization mode as: - * ``none``: write-ahead log is not maintained. - A node with ``wal_mode = none`` can't be replication master; - * ``write``: :ref:`fibers ` wait for their data to be written to - the write-ahead log (no :manpage:`fsync(2)`); - * ``fsync``: fibers wait for their data, :manpage:`fsync(2)` - follows each :manpage:`write(2)`; + * ``none``: write-ahead log is not maintained. + A node with ``wal_mode`` set to ``none`` can't be a replication master. + + * ``write``: :ref:`fibers ` wait for their data to be written to + the write-ahead log (no :manpage:`fsync(2)`). + + * ``fsync``: fibers wait for their data, :manpage:`fsync(2)` + follows each :manpage:`write(2)`. | Type: string | Default: "write" @@ -91,7 +93,7 @@ Since version 1.6.2. - Number of seconds between periodic scans of the write-ahead-log + The time interval in seconds between periodic scans of the write-ahead-log file directory, when checking for changes to write-ahead-log files for the sake of :ref:`replication ` or :ref:`hot standby `. @@ -128,20 +130,19 @@ Since version :doc:`2.6.3 `. - The delay (in seconds) used to prevent the :ref:`Tarantool garbage collector ` - from immediately removing :ref:`write-ahead log` files after a node restart. + The delay in seconds used to prevent the :ref:`Tarantool garbage collector ` + from immediately removing :ref:`write-ahead log ` files after a node restart. This delay eliminates possible erroneous situations when the master deletes WALs needed by :ref:`replicas ` after restart. As a consequence, replicas sync with the master faster after its restart and don't need to download all the data again. - - Once all the nodes in the replica set are up and running, - automatic cleanup is started again even if ``wal_cleanup_delay`` has not expired. + Once all the nodes in the replica set are up and running, a scheduled garbage collection is started again + even if ``wal_cleanup_delay`` has not expired. .. NOTE:: The ``wal_cleanup_delay`` option has no effect on nodes running as - :ref:`anonymous replicas`. + :ref:`anonymous replicas `. | Type: number | Default: 14400 seconds @@ -190,7 +191,7 @@ Note that records with additional fields are :ref:`replicated ` as follows: * If a replica doesn't support the extended format configured on a master, auxiliary fields are skipped. - * If a replica and master have different configurations for WAL records, a master's configuration is ignored. + * If a replica and master have different configurations for WAL records, the master's configuration is ignored. | Type: map | Default: nil diff --git a/doc/reference/configuration/cfg_snapshot_daemon.rst b/doc/reference/configuration/cfg_snapshot_daemon.rst index 0f24fc4495..adfa864771 100644 --- a/doc/reference/configuration/cfg_snapshot_daemon.rst +++ b/doc/reference/configuration/cfg_snapshot_daemon.rst @@ -2,138 +2,146 @@ * :ref:`checkpoint_interval ` * :ref:`checkpoint_wal_threshold ` -The checkpoint daemon is a fiber which is constantly running. At intervals, -it may make new :ref:`snapshot (.snap) files ` and then -may delete old snapshot files. +.. _cfg_checkpoint_daemon: -The :ref:`checkpoint_interval ` and -:ref:`checkpoint_count ` configuration -settings determine how long the intervals are, and how many snapshots should -exist before deletions occur. +**Checkpoint daemon** -.. _cfg_checkpoint_daemon-garbage-collector: +The checkpoint daemon (snapshot daemon) is a constantly running :ref:`fiber `. +The checkpoint daemon creates a schedule for the periodic snapshot creation based on +the configuration options and the speed of file size growth. +If enabled, the daemon makes new :ref:`snapshot ` (``.snap``) files according to this schedule. + +The work of the checkpoint daemon is based on the following configuration options: + +* :ref:`checkpoint_interval ` -- a new snapshot is taken once in a given period. +* :ref:`checkpoint_wal_threshold ` -- a new snapshot is taken once the size + of all WAL files created since the last snapshot exceeds a given limit. + +If necessary, the checkpoint daemon also activates the :ref:`Tarantool garbage collector ` +that deletes old snapshots and WAL files. + +.. _cfg_checkpoint_daemon-garbage-collector: **Tarantool garbage collector** -The checkpoint daemon may activate the Tarantool garbage collector -which deletes old files. This garbage collector is distinct from the -`Lua garbage collector `_ -which is for Lua objects, and distinct from a -Tarantool garbage collector which specializes in -:ref:`handling shard buckets `. +Tarantool garbage collector can be activated by the :ref:`checkpoint daemon `. +The garbage collector tracks the snapshots that are to be :ref:`relayed to a replica ` or needed +by other consumers. When the files are no longer needed, Tarantool garbage collector deletes them. + +.. NOTE:: + + The garbage collector called by the checkpoint daemon is distinct from the `Lua garbage collector `_ + which is for Lua objects, and distinct from the Tarantool garbage collector that specializes in :ref:`handling shard buckets `. + +This garbage collector is called as follows: + +* When the number of snapshots reaches the limit of :ref:`checkpoint_count ` size. + After a new snapshot is taken, Tarantool garbage collector deletes the oldest snapshot file and any associated WAL files. -If the checkpoint daemon deletes an old snapshot file, then the -Tarantool garbage collector will also delete -any :ref:`write-ahead log (.xlog) ` files which are older than -the snapshot file and which contain information that is present in the snapshot -file. It will also delete obsolete vinyl ``.run`` files. +* When the size of all WAL files created since the last snapshot reaches the limit of :ref:`checkpoint_wal_threshold `. + Once this size is exceeded, the checkpoint daemon takes a snapshot, then the garbage collector deletes the old WAL files. -The checkpoint daemon and the Tarantool garbage collector will not delete a file if: +If an old snapshot file is deleted, the Tarantool garbage collector also deletes +any :ref:`write-ahead log (.xlog) ` files that meet the following conditions: -* a **backup** is ongoing and the file has not been backed up - (see :ref:`"Hot backup" `), or +* The WAL files are older than the snapshot file. +* The WAL files contain information present in the snapshot file. -* **replication** is ongoing and the file has not been relayed to a replica - (see :ref:`"Replication architecture" `), +Tarantool garbage collector also deletes obsolete vinyl ``.run`` files. -* a replica is connecting, or +Tarantool garbage collector doesn't delete a file in the following cases: -* a replica has fallen behind. - The progress of each replica is tracked; if a replica's position is far - from being up to date, then the server stops to give it a chance to - catch up. - If an administrator concludes that a replica is permanently down, then the - correct procedure is to restart the server, or (preferably) - :ref:`remove the replica from the cluster `. +* A backup is running, and the file has not been backed up + (see :ref:`Hot backup `). -.. _cfg_checkpoint_daemon-checkpoint_interval: +* Replication is running, and the file has not been relayed to a replica + (see :ref:`Replication architecture `), -.. confval:: checkpoint_interval +* A replica is connecting. + +* A replica has fallen behind. + The progress of each replica is tracked; if a replica's position is far + from being up to date, then the server stops to give it a chance to catch up. + If an administrator concludes that a replica is permanently down, then the + correct procedure is to restart the server, or (preferably) :ref:`remove the replica from the cluster `. + +.. _cfg_checkpoint_daemon-checkpoint_interval: + +.. confval:: checkpoint_interval Since version 1.7.4. - The interval between actions by the checkpoint daemon, in seconds. If - ``checkpoint_interval`` is set to a value greater than zero, and there is - activity which causes change to a database, then the checkpoint daemon will - call :doc:`box.snapshot() ` every ``checkpoint_interval`` - seconds, creating a new snapshot file each time. If ``checkpoint_interval`` - is set to zero, then the checkpoint daemon is disabled. + The interval in seconds between actions by the :ref:`checkpoint daemon `. + If the option is set to a value greater than zero, and there is + activity that causes change to a database, then the checkpoint daemon + calls :doc:`box.snapshot() ` every ``checkpoint_interval`` + seconds, creating a new snapshot file each time. If the option + is set to zero, the checkpoint daemon is disabled. - For example: + **Example** - .. code-block:: lua + .. code-block:: lua - box.cfg{checkpoint_interval=60} + box.cfg{ checkpoint_interval = 7200 } - will cause the checkpoint daemon to create a new database snapshot once - per minute, if there is activity. + In the example, the checkpoint daemon creates a new database snapshot every two hours, if there is activity. | Type: integer | Default: 3600 (one hour) | Environment variable: TT_CHECKPOINT_INTERVAL | Dynamic: yes -.. _cfg_checkpoint_daemon-checkpoint_count: +.. _cfg_checkpoint_daemon-checkpoint_count: -.. confval:: checkpoint_count +.. confval:: checkpoint_count Since version 1.7.4. - The maximum number of snapshots that may exist on the - :ref:`memtx_dir ` directory - before the checkpoint daemon will delete old snapshots. - If ``checkpoint_count`` equals zero, then the checkpoint daemon - does not delete old snapshots. For example: + The maximum number of snapshots that are stored in the + :ref:`memtx_dir ` directory. + If the number of snapshots after creating a new one exceeds this value, + the :ref:`Tarantool garbage collector ` deletes old snapshots. + If the option is set to zero, the garbage collector + does not delete old snapshots. + + **Example** - .. code-block:: lua + .. code-block:: lua box.cfg{ - checkpoint_interval = 3600, - checkpoint_count = 10 + checkpoint_interval = 7200, + checkpoint_count = 3 } - will cause the checkpoint daemon to create a new snapshot each hour until - it has created ten snapshots. After that, it will delete the oldest snapshot - (and any associated write-ahead-log files) after creating a new one. + In the example, the checkpoint daemon creates a new snapshot every two hours until + it has created three snapshots. After creating a new snapshot (the fourth one), the oldest snapshot + and any associated write-ahead-log files are deleted. + + .. NOTE:: + + Snapshots will not be deleted if replication is ongoing and the file has not been relayed to a replica. + Therefore, ``checkpoint_count`` has no effect unless all replicas are alive. - Remember that, as noted earlier, snapshots will not be deleted if - replication is ongoing and the file has not been relayed to a replica. - Therefore ``checkpoint_count`` has no effect unless all replicas are alive. | Type: integer | Default: 2 | Environment variable: TT_CHECKPOINT_COUNT | Dynamic: yes -.. _cfg_checkpoint_daemon-checkpoint_wal_threshold: +.. _cfg_checkpoint_daemon-checkpoint_wal_threshold: -.. confval:: checkpoint_wal_threshold +.. confval:: checkpoint_wal_threshold Since version 2.1.2. - The threshold for the total size in bytes of all WAL files created since the last checkpoint. + The threshold for the total size in bytes for all WAL files created since the last checkpoint. Once the configured threshold is exceeded, the WAL thread notifies the - checkpoint daemon that it must make a new checkpoint and delete old WAL files. + :ref:`checkpoint daemon ` that it must make a new checkpoint and delete old WAL files. This parameter enables administrators to handle a problem that could occur with calculating how much disk space to allocate for a partition containing WAL files. - For example, suppose - :ref:`checkpoint_interval ` - = 2 and - :ref:`checkpoint_count ` - = 5 - and the average amount that Tarantool writes between each checkpoint interval - = 1 GB. - Then one could calculate that the necessary amount is (2*5*1) 10GB. - But this calculation would be wrong if, instead of writing 1 GB - during one checkpoint interval, - Tarantool encounters an unusual spike and tries to write 11 GB, - causing an operating-system ENOSPC ("no space") error. - By setting ``checkpoint_wal_threshold`` to a lower value, say 9 GB, - an administrator could prevent the error. - | Type: integer | Default: 10^18 (a large number so in effect there is no limit by default) | Environment variable: TT_CHECKPOINT_WAL_THRESHOLD diff --git a/doc/reference/configuration/configuration_reference.rst b/doc/reference/configuration/configuration_reference.rst index 791a7dc50d..08ea8b8c8b 100644 --- a/doc/reference/configuration/configuration_reference.rst +++ b/doc/reference/configuration/configuration_reference.rst @@ -2389,3 +2389,339 @@ The ``security`` section defines configuration parameters related to various sec | Type: boolean | Default: false | Environment variable: TT_SECURITY_SECURE_ERASING + +.. _configuration_reference_snapshot: + +snapshot +-------- + +The ``snapshot`` section defines configuration parameters related to the :ref:`snapshot files `. +To learn more about the snapshots' configuration, check the :ref:`Persistence ` page. + +.. NOTE:: + + ``snapshot`` can be defined in any :ref:`scope `. + +- :ref:`snapshot.dir ` +- :ref:`snapshot.snap_io_rate_limit ` +- :ref:`snapshot.count ` +- :ref:`snapshot.by.* ` + + - :ref:`snapshot.by.interval ` + - :ref:`snapshot.by.wal_size ` + +.. _configuration_reference_snapshot_dir: + +.. confval:: snapshot.dir + + A directory where memtx stores snapshot (.snap) files. + A relative path in this option is interpreted as relative to ``process.work_dir``. + + By default, snapshots and WAL files are stored in the same directory. + However, you can set different values for the ``snapshot.dir`` and :ref:`wal.dir ` options + to store them on different physical disks for performance matters. + + | + | Type: string + | Default: 'var/lib/{{ instance_name }}' + | Environment variable: TT_SNAPSHOT_DIR + +.. _configuration_reference_snapshot_snap_io_rate_limit: + +.. confval:: snapshot.snap_io_rate_limit + + Reduce the throttling effect of :doc:`box.snapshot() ` on + INSERT/UPDATE/DELETE performance by setting a limit on how many + megabytes per second it can write to disk. The same can be + achieved by splitting :ref:`wal.dir ` and + :ref:`snapshot.dir ` + locations and moving snapshots to a separate disk. + The limit also affects what + :ref:`box.stat.vinyl().regulator ` + may show for the write rate of dumps to ``.run`` and ``.index`` files. + + | + | Type: number + | Default: box.NULL + | Environment variable: TT_SNAPSHOT_SNAP_IO_RATE_LIMIT + +.. _configuration_reference_snapshot_count: + +.. confval:: snapshot.count + + The maximum number of snapshots that are stored in the + :ref:`snapshot.dir ` directory. + If the number of snapshots after creating a new one exceeds this value, + the :ref:`Tarantool garbage collector ` deletes old snapshots. + If ``snapshot.count`` is set to zero, the garbage collector + does not delete old snapshots. + + **Example** + + In the example, the checkpoint daemon creates a snapshot every two hours until + it has created three snapshots. After creating a new snapshot (the fourth one), the oldest snapshot + and any associated write-ahead-log files are deleted. + + .. code-block:: yaml + + snapshot: + by: + interval: 7200 + count: 3 + + .. NOTE:: + + Snapshots will not be deleted if replication is ongoing and the file has not been relayed to a replica. + Therefore, ``snapshot.count`` has no effect unless all replicas are alive. + + | + | Type: integer + | Default: 2 + | Environment variable: TT_SNAPSHOT_COUNT + +.. _configuration_reference_snapshot_by: + +snapshot.by.* +~~~~~~~~~~~~~ + +.. _configuration_reference_snapshot_by_interval: + +.. confval:: snapshot.by.interval + + The interval in seconds between actions by the :ref:`checkpoint daemon `. + If the option is set to a value greater than zero, and there is + activity that causes change to a database, then the checkpoint daemon calls + :doc:`box.snapshot() ` every ``snapshot.by.interval`` + seconds, creating a new snapshot file each time. + If the option is set to zero, the checkpoint daemon is disabled. + + **Example** + + In the example, the checkpoint daemon creates a new database snapshot every two hours, if there is activity. + + .. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_snapshot/config.yaml + :language: yaml + :start-at: by: + :end-at: interval: 7200 + :dedent: + + | + | Type: number + | Default: 3600 + | Environment variable: TT_SNAPSHOT_BY_INTERVAL + +.. _configuration_reference_snapshot_by_wal_size: + +.. confval:: snapshot.by.wal_size + + The threshold for the total size in bytes for all WAL files created since the last snapshot taken. + Once the configured threshold is exceeded, the WAL thread notifies the + :ref:`checkpoint daemon ` that it must make a new snapshot and delete old WAL files. + + | + | Type: integer + | Default: 10^18 + | Environment variable: TT_SNAPSHOT_BY_WAL_SIZE + +.. _configuration_reference_wal: + +wal +--- + +The ``wal`` section defines configuration parameters related to :ref:`write-ahead log `. +To learn more about the WAL configuration, check the :ref:`Persistence ` page. + +.. NOTE:: + + ``wal`` can be defined in any :ref:`scope `. + +- :ref:`wal.cleanup_delay ` +- :ref:`wal.dir ` +- :ref:`wal.dir_rescan_delay ` +- :ref:`wal.max_size ` +- :ref:`wal.mode ` +- :ref:`wal.queue_max_size ` +- :ref:`wal.ext.* ` + + - :ref:`wal.ext.new ` + - :ref:`wal.ext.old ` + - :ref:`wal.ext.spaces ` + +.. _configuration_reference_wal_cleanup_delay: + +.. confval:: wal.cleanup_delay + + The delay in seconds used to prevent the :ref:`Tarantool garbage collector ` + from immediately removing :ref:`write-ahead log ` files after a node restart. + This delay eliminates possible erroneous situations when the master deletes WALs + needed by :ref:`replicas ` after restart. + As a consequence, replicas sync with the master faster after its restart and + don't need to download all the data again. + Once all the nodes in the replica set are up and running, a scheduled garbage collection is started again + even if ``wal.cleanup_delay`` has not expired. + + + .. NOTE:: + + The option has no effect on nodes running as + :ref:`anonymous replicas `. + + | + | Type: number + | Default: 14400 + | Environment variable: TT_WAL_CLEANUP_DELAY + +.. _configuration_reference_wal_dir: + +.. confval:: wal.dir + + A directory where write-ahead log (``.xlog``) files are stored. + A relative path in this option is interpreted as relative to ``process.work_dir``. + + By default, WAL files and snapshots are stored in the same directory. + However, you can set different values for the ``wal.dir`` and :ref:`snapshot.dir ` options + to store them on different physical disks for performance matters. + + | + | Type: string + | Default: 'var/lib/{{ instance_name }}' + | Environment variable: TT_WAL_DIR + +.. _configuration_reference_wal_dir_rescan_delay: + +.. confval:: wal.dir_rescan_delay + + The time interval in seconds between periodic scans of the write-ahead-log + file directory, when checking for changes to write-ahead-log + files for the sake of :ref:`replication ` or :ref:`hot standby `. + + | + | Type: number + | Default: 2 + | Environment variable: TT_WAL_DIR_RESCAN_DELAY + +.. _configuration_reference_wal_max_size: + +.. confval:: wal.max_size + + The maximum number of bytes in a single write-ahead log file. + When a request would cause an ``.xlog`` file to become larger than + ``wal.max_size``, Tarantool creates a new WAL file. + + | + | Type: integer + | Default: 268435456 + | Environment variable: TT_WAL_MAX_SIZE + +.. _configuration_reference_wal_mode: + +.. confval:: wal.mode + + Specify fiber-WAL-disk synchronization mode as: + + * ``none``: write-ahead log is not maintained. + A node with ``wal.mode`` set to ``none`` can't be a replication master. + + * ``write``: :ref:`fibers ` wait for their data to be written to + the write-ahead log (no ``fsync(2)``). + + * ``fsync``: fibers wait for their data, :manpage:`fsync(2)` + follows each :manpage:`write(2)`. + + | + | Type: string + | Default: 'write' + | Environment variable: TT_WAL_MODE + +.. _configuration_reference_wal_queue_max_size: + +.. confval:: wal.queue_max_size + + The size of the queue in bytes used by a :ref:`replica ` to submit + new transactions to a :ref:`write-ahead log ` (WAL). + This option helps limit the rate at which a replica submits transactions to the WAL. + + Limiting the queue size might be useful when a replica is trying to sync with a master and + reads new transactions faster than writing them to the WAL. + + .. NOTE:: + + You might consider increasing the ``wal.queue_max_size`` value in case of + large tuples (approximately one megabyte or larger). + + | + | Type: integer + | Default: 16777216 + | Environment variable: TT_WAL_QUEUE_MAX_SIZE + +.. _configuration_reference_wal_ext: + +wal.ext.* +~~~~~~~~~ + +.. admonition:: Enterprise Edition + :class: fact + + Configuring ``wal.ext.*`` parameters is available in the `Enterprise Edition `_ only. + +This section describes options related to :ref:`WAL extensions `. + +.. _configuration_reference_wal_ext_new: + +.. confval:: wal.ext.new + + Enable storing a new tuple for each :ref:`CRUD ` operation performed. + The option is in effect for all spaces. + To adjust the option for specific spaces, use the :ref:`wal.ext.spaces ` + option. + + | + | Type: boolean + | Default: false + | Environment variable: TT_WAL_EXT_NEW + +.. _configuration_reference_wal_ext_old: + +.. confval:: wal.ext.old + + Enable storing an old tuple for each :ref:`CRUD ` operation performed. + The option is in effect for all spaces. + To adjust the option for specific spaces, use the :ref:`wal.ext.spaces ` + option. + + | + | Type: boolean + | Default: false + | Environment variable: TT_WAL_EXT_OLD + +.. _configuration_reference_wal_ext_spaces: + +.. confval:: wal.ext.spaces + + Enable or disable storing an old and new tuple in the :ref:`WAL ` record + for a given space explicitly. + The configuration for specific spaces has priority over the configuration in the + :ref:`wal.ext.new ` and :ref:`wal.ext.old ` + options. + + The option is a key-value pair: + + * The key is a space name (string). + + * The value is a table that includes two optional boolean options: ``old`` and ``new``. + The format and the default value of these options are described in ``wal.ext.old`` and ``wal.ext.new``. + + **Example** + + In the example, only new tuples are added to the log for the ``bands`` space. + + .. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml + :language: yaml + :start-at: ext: + :end-at: old: false + :dedent: + + | + | Type: map + | Default: nil + | Environment variable: TT_WAL_EXT_SPACES diff --git a/doc/reference/reference_lua/box_snapshot.rst b/doc/reference/reference_lua/box_snapshot.rst index 01e91c524b..9bcf3d9dd1 100644 --- a/doc/reference/reference_lua/box_snapshot.rst +++ b/doc/reference/reference_lua/box_snapshot.rst @@ -9,7 +9,7 @@ **Memtx** Take a snapshot of all data and store it in - :ref:`memtx_dir `:samp:`/{}.snap`. + :ref:`snapshot.dir `:samp:`/{}.snap`. To take a snapshot, Tarantool first enters the delayed garbage collection mode for all data. In this mode, the :ref:`Tarantool garbage collector ` @@ -26,7 +26,7 @@ performance (averaging to 80MB/second on modern disks), which means an average database instance gets saved in a matter of minutes. You may restrict the speed by changing - :ref:`snap_io_rate_limit `. + :ref:`snapshot.snap_io_rate_limit `. .. NOTE:: @@ -46,7 +46,7 @@ Although ``box.snapshot()`` does not cause a fork, there is a separate fiber which may produce snapshots at regular intervals -- see the discussion of - the :ref:`checkpoint daemon `. + the :ref:`checkpoint daemon `. **Example:**