Skip to content

PS-10190 [DOCS] - Binary Log UDFs functions documentation incomplete 8.4 #571

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: 8.4
Choose a base branch
from
Open

PS-10190 [DOCS] - Binary Log UDFs functions documentation incomplete 8.4 #571

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
197 changes: 135 additions & 62 deletions docs/binlogging-replication-improvements.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,139 +165,212 @@ mysql> SET binlog_ddl_skip_rewrite = ON;

## Point-in-Time Recovery with `binlog_utils_udf`

Use the binlog_utils_udf component to assist with Point-in-Time Recovery (PiTR).
The component installs user-defined functions (UDFs) that help you map GTIDs to
binary log files and inspect the contents and timestamps of binlog files.
Point-in-Time Recovery (PiTR) allows you to restore a database to any specific moment in time using binary logs. The `binlog_utils_udf` component provides user-defined functions (UDFs) that simplify PiTR operations by helping you:

### Functions
* Map Global Transaction Identifiers (GTIDs) to specific binary log files
* Inspect binary log contents and timestamps
* Locate the exact binary log files needed for recovery operations

| Function | Returns | Description |
|-----------------------------------------|----------------------|-------------------------------------------------------------------------|
| get_binlog_by_gtid(gtid) | STRING (binlog name) | Returns the binlog file that contains the specified GTID. |
| get_last_gtid_from_binlog(binlog) | STRING (GTID) | Returns the last GTID found in the specified binlog. |
| get_gtid_set_by_binlog(binlog) | STRING (GTID set) | Returns all GTIDs found in the specified binlog. |
| get_binlog_by_gtid_set(gtid_set) | STRING (binlog name) | Returns the first binlog file that contains at least one GTID from the specified set. |
| get_first_record_timestamp_by_binlog(binlog) | INTEGER (timestamp) | Returns the timestamp of the first event in the specified binlog. |
| get_last_record_timestamp_by_binlog(binlog) | INTEGER (timestamp) | Returns the timestamp of the last event in the specified binlog. |
These functions are particularly useful when you need to determine which binary log files contain specific transactions or events during recovery planning.

### Notes
### Prerequisites

* Timestamp-returning functions provide values with microsecond precision in
UNIX time. Each value represents the number of microseconds since
1970-01-01 00:00:00 UTC.
Before using the `binlog_utils_udf` component, ensure the following requirements are met:

* Functions that accept a binlog name require a short file name only. Do not
include a path. If the input contains a path separator (/), the server
returns an error.
* Percona Server for MySQL: The component is only available in Percona Server for MySQL, not in standard MySQL

* The server reads binlogs from the current binlog directory defined by the
@@log_bin_basename system variable.
* Binary logging enabled: The server must have binary logging enabled (`log_bin` system variable set to `ON`)

* Functions that return a binlog file name return the short name (no path).
* GTID enabled: For GTID-related functions, GTID must be enabled (`gtid_mode` set to `ON`)

### Install the component
* MySQL privileges: You need `SYSTEM_VARIABLES_ADMIN` privilege to install components. For binary log operations, `BINLOG_ADMIN` privilege may also be required. The `SUPER` privilege is deprecated in MySQL 8.0+ and should be replaced with specific dynamic privileges

Install the component once on each server where you want to use these UDFs.
#### Install the component

Install the component on each server where you plan to use these functions:

```{.bash data-prompt="mysql>"}
mysql> INSTALL COMPONENT 'file://component_binlog_utils_udf';
```

You can confirm installation by checking the list of registered functions:
#### Verify installation

Confirm the component is loaded successfully:

```{.bash data-prompt="mysql>"}
mysql> SELECT * FROM mysql.component WHERE component_urn = 'file://component_binlog_utils_udf';
```

The query should return one row if the component is installed. You can also verify by checking for the available functions:

```{.bash data-prompt="mysql>"}
mysql> SELECT ROUTINE_NAME FROM INFORMATION_SCHEMA.ROUTINES \G
mysql> SELECT ROUTINE_NAME FROM INFORMATION_SCHEMA.ROUTINES
-> WHERE ROUTINE_NAME LIKE 'get_%' AND ROUTINE_TYPE = 'FUNCTION';
```

### Available functions

The `binlog_utils_udf` component provides six functions for binary log analysis and GTID mapping:

| Function | Returns | Description | Use Case |
|-----------------------------------------|----------------------|-------------------------------------------------------------------------|----------|
| [`get_binlog_by_gtid(gtid)`](#find-binary-log-by-gtid) | STRING (binlog name) | Returns the binary log file that contains the specified GTID. | Find which binary log contains a specific transaction |
| [`get_last_gtid_from_binlog(binlog)`](#get-last-gtid-from-binary-log) | STRING (GTID) | Returns the last GTID found in the specified binary log. | Identify the final transaction in a binary log file |
| [`get_gtid_set_by_binlog(binlog)`](#get-all-gtids-from-binary-log) | STRING (GTID set) | Returns all GTIDs found in the specified binary log. | Get complete list of transactions in a binary log |
| [`get_binlog_by_gtid_set(gtid_set)`](#find-binary-log-by-gtid-set) | STRING (binlog name) | Returns the first binary log file that contains at least one GTID from the specified set. | Find binary log containing any transaction from a GTID set |
| [`get_first_record_timestamp_by_binlog(binlog)`](#get-first-event-timestamp) | INTEGER (timestamp) | Returns the timestamp of the first event in the specified binary log. | Determine when a binary log file started |
| [`get_last_record_timestamp_by_binlog(binlog)`](#get-last-event-timestamp) | INTEGER (timestamp) | Returns the timestamp of the last event in the specified binary log. | Determine when a binary log file ended |

### Important notes

* CAST requirement: When using these user-defined functions, you must use CAST to return a result. String functions require `CAST(...AS CHAR)` and timestamp functions require `CAST(...AS UNSIGNED)`.

* Timestamp precision: Timestamp-returning functions provide values with microsecond precision in UNIX time format. Each value represents the number of microseconds since 1970-01-01 00:00:00 UTC.

* Binary log file names: Functions that accept a binary log name require only the short file name (for example, `binlog.000001`). Do not include the full path. If the input contains a path separator (`/`), the server returns an error.

* Binary log directory: The server reads binary logs from the directory defined by the `@@log_bin_basename` system variable.

* Return values: Functions that return binary log file names return only the short name without the path.

* Performance considerations: These functions read binary log files directly from disk. For large binary log files, the functions may take several seconds to complete.



### Usage examples

Replace the sample arguments with values from your environment. The examples
show the typical way to call each function. For clarity, results are aliased.
The following examples demonstrate how to use each function. Replace the sample arguments with values from your environment. All examples include the required CAST statements for proper function execution.

#### get_binlog_by_gtid()
#### Find binary log by GTID

Locate the binlog that contains a GTID:
Use `get_binlog_by_gtid()` to locate which binary log file contains a specific transaction:

```{.bash data-prompt="mysql>"}
mysql> SELECT get_binlog_by_gtid('UUID-GROUP:1') AS binlog;
mysql> SELECT CAST(get_binlog_by_gtid('550e8400-e29b-41d4-a716-446655440000:123') AS CHAR) AS binlog;
```

#### get_last_gtid_from_binlog()
Use case: When you know a specific GTID and need to find which binary log file contains that transaction for recovery purposes.

#### Get last GTID from binary log

Return the last GTID in a binlog
Use `get_last_gtid_from_binlog()` to find the final transaction in a specific binary log file:

```{.bash data-prompt="mysql>"}
mysql> SELECT get_last_gtid_from_binlog('binlog.000001') AS last_gtid;
mysql> SELECT CAST(get_last_gtid_from_binlog('binlog.000001') AS CHAR) AS last_gtid;
```

#### get_gtid_set_by_binlog()
Use case: Determine the last transaction processed in a binary log file before rotating to the next file.

Return all GTIDs in a binlog
#### Get all GTIDs from binary log

Use `get_gtid_set_by_binlog()` to retrieve all GTIDs contained in a specific binary log file:

```{.bash data-prompt="mysql>"}
mysql> SELECT get_gtid_set_by_binlog('binlog.000001') AS gtid_set;
mysql> SELECT CAST(get_gtid_set_by_binlog('binlog.000001') AS CHAR) AS gtid_set;
```

#### get_binlog_by_gtid_set()
Use case: Get a complete list of all transactions in a binary log file for analysis or replication setup.

#### Find binary log by GTID set

Find a binlog that contains any GTID in a set
Use `get_binlog_by_gtid_set()` to find the first binary log file that contains any GTID from a specified set:

```{.bash data-prompt="mysql>"}
mysql> SELECT get_binlog_by_gtid_set('UUID1:7,UUID1:8') AS binlog;
mysql> SELECT CAST(get_binlog_by_gtid_set('550e8400-e29b-41d4-a716-446655440000:7,550e8400-e29b-41d4-a716-446655440000:8') AS CHAR) AS binlog;
```

#### get_first_record_timestamp_by_binlog() and get_last_record_timestamp_by_binlog(binlog)
Use case: When you have a set of GTIDs and need to find which binary log file contains at least one of those transactions.

#### Get binary log timestamps

Get the first event timestamp from a binlog. The function returns microseconds since the UNIX epoch. Use the tabs below to
see the raw numeric value or a human-readable timestamp.
Use timestamp functions to determine when events occurred in binary log files. These functions return microsecond-precision timestamps in UNIX time format.

=== "Raw Timestamp"
##### Get first event timestamp

Find when the first event was written to a binary log file:

=== "Raw Timestamp (Microseconds)"
```{.bash data-prompt="mysql>"}
mysql> SELECT get_first_record_timestamp_by_binlog('binlog.000001') AS raw_ts;
mysql> SELECT CAST(get_first_record_timestamp_by_binlog('binlog.000001') AS UNSIGNED) AS raw_ts;
```

=== "Human-Readable"
=== "Human-Readable Format"
```{.bash data-prompt="mysql>"}
mysql> SELECT FROM_UNIXTIME(
get_first_record_timestamp_by_binlog('binlog.000001') DIV 1000000
CAST(get_first_record_timestamp_by_binlog('binlog.000001') AS UNSIGNED) DIV 1000000
) AS first_event_ts;
```

Get the last event timestamp from a binlog
Use case: Determine when a binary log file started receiving events, useful for recovery planning.

##### Get last event timestamp

=== "Raw Timestamp"
Find when the last event was written to a binary log file:

=== "Raw Timestamp (Microseconds)"
```{.bash data-prompt="mysql>"}
mysql> SELECT get_last_record_timestamp_by_binlog('binlog.000001') AS raw_ts;
mysql> SELECT CAST(get_last_record_timestamp_by_binlog('binlog.000001') AS UNSIGNED) AS raw_ts;
```

=== "Human-Readable"
=== "Human-Readable Format"
```{.bash data-prompt="mysql>"}
mysql> SELECT FROM_UNIXTIME(
get_last_record_timestamp_by_binlog('binlog.000001') DIV 1000000
CAST(get_last_record_timestamp_by_binlog('binlog.000001') AS UNSIGNED) DIV 1000000
) AS last_event_ts;
```

Use case: Determine when a binary log file stopped receiving events, useful for understanding binary log rotation timing.


### Troubleshooting

#### Common issues

Function returns NULL: This usually indicates that the specified GTID or binary log file does not exist. Verify that:

* The GTID format is correct (UUID:transaction_id)

??? example "Expected output"
* The binary log file exists in the binary log directory

```{.text .no-copy}
+---------------+
| binlog |
+---------------+
| binlog.000001 |
+---------------+
```
* GTID is enabled on the server

Actual values depend on your server state and binlog contents.
Error: "Unknown function": The component is not installed. Install the component using the `INSTALL COMPONENT` command.

Error: "Access denied": You need `SYSTEM_VARIABLES_ADMIN` privilege to install the component and `BINLOG_ADMIN` privilege for binary log operations. The `SUPER` privilege is deprecated in MySQL 8.0+.

Performance issues: These functions read binary log files directly from disk. For large binary log files, expect execution times of several seconds.

#### Verify binary log files

Check which binary log files are available:

```{.bash data-prompt="mysql>"}
mysql> SHOW BINARY LOGS;
```

#### Check GTID status

Verify GTID is enabled:

```{.bash data-prompt="mysql>"}
mysql> SHOW VARIABLES LIKE 'gtid_mode';
```

### Uninstall the component

Remove the component and all associated UDFs:
Remove the component and all associated functions:

```{.bash data-prompt="mysql>"}
mysql> UNINSTALL COMPONENT 'file://component_binlog_utils_udf';
```

Verify removal:

```{.bash data-prompt="mysql>"}
mysql> SELECT * FROM mysql.component WHERE component_urn = 'file://component_binlog_utils_udf';
```

The query should return no rows if the component is successfully uninstalled.

## Limitations

Expand Down