From 294dcf016cd14fc84f4240956166482468426c9b Mon Sep 17 00:00:00 2001 From: James Wheare Date: Mon, 23 Sep 2019 16:55:51 +0100 Subject: [PATCH 01/13] [multiline] Multiline message batches --- extensions/multiline.md | 237 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 237 insertions(+) create mode 100644 extensions/multiline.md diff --git a/extensions/multiline.md b/extensions/multiline.md new file mode 100644 index 000000000..64a844394 --- /dev/null +++ b/extensions/multiline.md @@ -0,0 +1,237 @@ +--- +title: Multiline messages +layout: spec +work-in-progress: true +copyrights: + - + name: "James Wheare" + period: "2019" + email: "james@irccloud.com" +--- + +## Notes for implementing work-in-progress version + +This is a work-in-progress specification. + +Software implementing this work-in-progress specification MUST NOT use the unprefixed `multiline-concat` tag or `multiline` CAP/batch names. Instead, implementations SHOULD use the `draft/multiline-concat` tag, `draft/multiline` CAP and `draft/multiline` batch names to be interoperable with other software implementing a compatible work-in-progress version. + +The final version of the specification will use an unprefixed tag name. + +## Introduction + +This specification adds a new batch type and tag sent by clients and servers to send messages that can exceed the usual byte length limit and that can contain line breaks. + +## Motivation + +IRC messages have been traditionally limited by the line-based nature of the protocol and the line length limit of 512. To work around these limitations, client implementations split longer messages into multiple lines, and users need to rely on snippet hosting services to send multiple lines without flooding the chat or having their messages interrupted by other users. + +Mulitline messages allow for a more coherent experience that avoids the disconnected messages that result from these workarounds. + +## Architecture + +### Dependencies + +This specification uses the [batch][] capability which MUST be negotiated at the same time. It also uses the [message tags][] and [standard replies][] frameworks. + +### Capabilities + +This specification adds the `draft/multiline` capability. + +Implementations requesting this capability indicate that they are capable of handling the batch type and message tag described below. + +The capability has a REQUIRED value: a comma (`,`) (0x2C) separated list of tokens. Each token consists of a key which might have a value attached. If there is a value attached, the value is separated from the key by an equals sign (`=`) (0x3D). That is, `[=][,[=][,[=]]]`. Keys specified in this document MUST only occur at most once. + +Clients MUST ignore every token with a key that they don't understand. + +The only defined capability key so far is: + +* `max-bytes` - This defines the maximum allowed total byte length of multiline batched content *REQUIRED* +* `max-lines` - This defines the maximum allowed number of lines in a multiline batch content *OPTIONAL* + +### Tags + +This specification adds the `draft/multiline-concat` message tag, which has no value and is used to send message lines that extend beyond the usual byte length limit. Its usage is described below. + +### Batch types + +This specification adds the `draft/multiline` batch type, and introduces client initiated batches. These use the same syntax as server initiated batches. + +In addition to the base batch parameters (reference-tag and type) a multiline batch has one additional parameter, the target recipient. + +Multiline batches MUST contain one or more PRIVMSG lines whose target MUST match the batch target. + +When receiving a well-formed mulitiline message batch, implementations MUST collect each PRIVMSG message content and wait until the full batch has been received before processing the message. Processing in this context refers to: + +* Servers: delivering the batch to the intended recipients +* Clients: displaying the batched message to the user + +Messages in a multiline batch MUST be concatenated with a line break unless the `draft/multiline-concat` message tag is sent, in which case the message is directly concatenated with the previous message without any separation. + +Clients MUST NOT send messages other than PRIVMSG while a multiline batch is open. + +### Fallback + +When delivering multiline batches to clients that have not negotiated the multiline capability, servers MUST deliver the component messages without using a multiline BATCH. + +Any tags that would have been added to the batch, e.g. message IDs, account tags etc MUST be included on the first message line. Tags MAY also be included on subsequent lines where it makes sense to do so. + +### Errors + +Servers MUST use `FAIL` messages from the [standard replies][] framework to notify clients of errors with multiline batches. The command is `BATCH` and the following codes are defined: + +* `MULTILINE_MAX_BYTES `: `max-bytes` limit exceeded +* `MULTILINE_MAX_LINES `: `max-lines` limit exceeded +* `MULTILINE_INVALID_TARGET `: mismatched PRIVMSG target within a batch +* `MULTILINE_INVALID`: any other error + +## Interaction with other specifications + +### Message tags ([spec][message tags]) + +Clients MUST only send non-`batch` tags on the opening BATCH command when sending multiline batches. + +### Message ids ([spec][message ids]) + +Servers MUST only include a message ID on the first message of a batch when sending a fallback to non-supporting clients. + +### Labeled response ([draft][labeled response]) with echo-message ([spec][echo message]) + +Servers MUST only include the label tag on the opening BATCH command when replying to a client using `echo-message`. + +## Client implementation considerations + +This section is non-normative. + +### Splitting long lines + +When splitting long lines, care is required to ensure messages are displayed appropriately when combined or in the fallback case. + +The recommended splitting method is to split long lines between words where possible and leave the space character at the end of the line. This ensures that concatenated lines don't lose the space, while fallback lines don't start with a space. + +In order to calculate the maximum allowed space before splitting a line, consider the following line: + +``` +:nick!~user@host PRIVMSG #channel :hello \n\r +``` + +The parts of a message that must fit within the 512 byte limit are as follows + +| part | syntax | length | +|-|-|-| +| leading | `:` | 1 | +| nick | `nick` | variable | +| nick/mask separator | `!` | 1 | +| user | `~user` | variable | +| user/host separator | `@` | 1 | +| host | `host` | variable | +| command | `PRIVMSG` | 7 | +| channel | `#channel` | variable | +| message separator | ` :` | 2 | +| crlf | `\n\r` | 2 | +| message | `hello ` | variable | + +Not all parts may appear in a message, but a safe limit should take them into account and include an extra safety buffer in case of any unexpected syntax added by a server, e.g. a statusmsg prefix. + +With a safety buffer of 10, the total message size available before a split is needed can be calculated as follows + +``` +512 - 14 - 10 - - - - +``` + +Clients might wish to perform a `WHO` query on themselves when connecting to a server, and keep track of numeric `396 RPL_VISIBLEHOST` to keep their `user@host` mask in sync. + +Clients might alternatively prefer to use a hard coded limit. A possible worst case scenario for the variable parts might be: + +* nick: 20 +* user: 20 +* host: 63 +* channel: 32 + +Which would leave 353 bytes for each message. + +## Server implementation considerations + +This section is non-normative. + +In the context of flood protection, keep in mind that clients might send an entire batched multiline message all at once. + +## Examples + +This section is non-normative. + +NOTE: In these examples, `` indicates a space character which would otherwise not be clearly visible. + +Client sending a mutliline batch + + Client: BATCH +123 draft/multiline #channel + Client: @batch=123 PRIVMSG #channel hello + Client: @batch=123 privmsg #channel :how is + Client: @batch=123;draft/multiline-concat PRIVMSG #channel :everyone? + Client: BATCH -123 + +Server sending the same batch + + Server: @msgid=xxx;account=account :n!u@h BATCH +123 draft/multiline #channel + Server: @batch=123 :n!u@h PRIVMSG #channel hello + Server: @batch=123 :n!u@h PRIVMSG #channel :how is + Server: @batch=123;draft/multiline-concat :n!u@h PRIVMSG #channel :everyone? + Server: BATCH -123 + +Server sending messages to clients without multiline support + + Server: @msgid=xxx;account=account :n!u@h PRIVMSG #channel hello + Server: @account=account :n!u@h PRIVMSG #channel :how is + Server: @account=account :n!u@h PRIVMSG #channel :everyone? + +Final concatenated message + + hello + how is everyone? + +--- + +Invalid multiline batch target + + Client: BATCH +456 draft/multiline #foo + Client: @batch=456 PRIVMSG #bar hello + Client: BATCH -456 + + Server: :irc.example.com FAIl BATCH MULTILINE_INVALID_TARGET #foo #bar :Invalid multiline target + +--- + +`max-bytes` exceeded multiline batch + + Server: CAP * LS :draft/multiline=max-bytes=40000 + + Client: BATCH +abc draft/multiline #channel + Client: @batch=abc PRIVMSG #channel hello + ... lines that exceeed the max-bytes=40000 limit ... + + Server: :irc.example.com FAIl BATCH MULTILINE_MAX_BYTES 40000 :Multiline batch max-bytes exceeded + +`max-bytes` exceeded multiline batch + + Server: CAP * LS :draft/multiline=max-lines=10 + + Client: BATCH +abc draft/multiline #channel + Client: @batch=abc PRIVMSG #channel hello + ... 10 more lines ... + + Server: :irc.example.com FAIl BATCH MULTILINE_MAX_LINES 10 :Multiline batch max-lines exceeded + +--- + +Invalid multiline batch + + Client: BATCH +999 draft/multiline #channel + ... invalid batch contents ... + + Server: :irc.example.com FAIl BATCH MULTILINE_INVALID :Invalid multiline batch + +[message tags]: ../extensions/message-tags.html +[batch]: ../extensions/batch-3.2.html +[standard replies]: ../extensions/standard-replies.html +[message ids]: ../extensions/message-ids.html +[labeled response]: ../extensions/labeled-response.html +[echo message]: ../extensions/echo-message-3.2.html From d09657ddbc438c7e7279bff59aabd449b30ffcdb Mon Sep 17 00:00:00 2001 From: James Wheare Date: Mon, 23 Sep 2019 17:39:25 +0100 Subject: [PATCH 02/13] [multiline] Add jesopo to contributors --- extensions/multiline.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/extensions/multiline.md b/extensions/multiline.md index 64a844394..7f23a691b 100644 --- a/extensions/multiline.md +++ b/extensions/multiline.md @@ -7,6 +7,10 @@ copyrights: name: "James Wheare" period: "2019" email: "james@irccloud.com" +contributors: + - + name: "jesopo" + email: "jess@jesopo.uk" --- ## Notes for implementing work-in-progress version From 65b2f96d5c1fbf9dc66bf8a5a6700bd193c2618f Mon Sep 17 00:00:00 2001 From: James Wheare Date: Fri, 17 Jan 2020 17:19:15 +0000 Subject: [PATCH 03/13] Fix max-lines example --- extensions/multiline.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/extensions/multiline.md b/extensions/multiline.md index 7f23a691b..a4670804c 100644 --- a/extensions/multiline.md +++ b/extensions/multiline.md @@ -214,9 +214,9 @@ Invalid multiline batch target Server: :irc.example.com FAIl BATCH MULTILINE_MAX_BYTES 40000 :Multiline batch max-bytes exceeded -`max-bytes` exceeded multiline batch +`max-lines` exceeded multiline batch - Server: CAP * LS :draft/multiline=max-lines=10 + Server: CAP * LS :draft/multiline=max-bytes=40000,max-lines=10 Client: BATCH +abc draft/multiline #channel Client: @batch=abc PRIVMSG #channel hello From a4c6f7e220b4fe94c963caba230b28bf4077a1ca Mon Sep 17 00:00:00 2001 From: James Wheare Date: Fri, 17 Jan 2020 17:20:45 +0000 Subject: [PATCH 04/13] Clarify tags on initial BATCH command requirement --- extensions/multiline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/extensions/multiline.md b/extensions/multiline.md index a4670804c..734aaeb7a 100644 --- a/extensions/multiline.md +++ b/extensions/multiline.md @@ -92,7 +92,7 @@ Servers MUST use `FAIL` messages from the [standard replies][] framework to noti ### Message tags ([spec][message tags]) -Clients MUST only send non-`batch` tags on the opening BATCH command when sending multiline batches. +Clients MUST NOT send tags other than `draft/multiline-concat` and `batch` on messages within the batch. In particular, all client-only tags associated with the message must be sent attached to the initial BATCH command. ### Message ids ([spec][message ids]) From 858e5faaf2e80d20056205a71f3a4c042e8d1993 Mon Sep 17 00:00:00 2001 From: James Wheare Date: Fri, 17 Jan 2020 17:24:36 +0000 Subject: [PATCH 05/13] Clarify splitting long lines section --- extensions/multiline.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/extensions/multiline.md b/extensions/multiline.md index 734aaeb7a..b9be5938b 100644 --- a/extensions/multiline.md +++ b/extensions/multiline.md @@ -108,9 +108,9 @@ This section is non-normative. ### Splitting long lines -When splitting long lines, care is required to ensure messages are displayed appropriately when combined or in the fallback case. +When using `draft/multiline-concat` to split longer lines, care is required to ensure messages are displayed appropriately when combined or in the fallback case. -The recommended splitting method is to split long lines between words where possible and leave the space character at the end of the line. This ensures that concatenated lines don't lose the space, while fallback lines don't start with a space. +The recommended splitting method is to split long lines between words where possible and leave the space character at the end of the line. This ensures that pairs of words spanning a line concatenation are still space separated, without adding a space to the start of any fallback lines. In order to calculate the maximum allowed space before splitting a line, consider the following line: From 5947faa852f15498dad607475597eeca7f9c5b70 Mon Sep 17 00:00:00 2001 From: James Wheare Date: Fri, 10 Apr 2020 16:41:10 +0100 Subject: [PATCH 06/13] [multiline] Fix CRLF refs --- extensions/multiline.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/extensions/multiline.md b/extensions/multiline.md index b9be5938b..5d4f749ee 100644 --- a/extensions/multiline.md +++ b/extensions/multiline.md @@ -115,7 +115,7 @@ The recommended splitting method is to split long lines between words where poss In order to calculate the maximum allowed space before splitting a line, consider the following line: ``` -:nick!~user@host PRIVMSG #channel :hello \n\r +:nick!~user@host PRIVMSG #channel :hello \r\n ``` The parts of a message that must fit within the 512 byte limit are as follows @@ -131,7 +131,7 @@ The parts of a message that must fit within the 512 byte limit are as follows | command | `PRIVMSG` | 7 | | channel | `#channel` | variable | | message separator | ` :` | 2 | -| crlf | `\n\r` | 2 | +| crlf | `\r\n` | 2 | | message | `hello ` | variable | Not all parts may appear in a message, but a safe limit should take them into account and include an extra safety buffer in case of any unexpected syntax added by a server, e.g. a statusmsg prefix. From 281f386f08c9c0e7b9fbff1c9f30eba8705fe615 Mon Sep 17 00:00:00 2001 From: James Wheare Date: Wed, 22 Apr 2020 10:47:37 +0100 Subject: [PATCH 07/13] Make max-lines RECOMMENDED instead of OPTIONAL --- extensions/multiline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/extensions/multiline.md b/extensions/multiline.md index 5d4f749ee..ad9858f51 100644 --- a/extensions/multiline.md +++ b/extensions/multiline.md @@ -50,7 +50,7 @@ Clients MUST ignore every token with a key that they don't understand. The only defined capability key so far is: * `max-bytes` - This defines the maximum allowed total byte length of multiline batched content *REQUIRED* -* `max-lines` - This defines the maximum allowed number of lines in a multiline batch content *OPTIONAL* +* `max-lines` - This defines the maximum allowed number of lines in a multiline batch content *RECOMMENDED* ### Tags From 0b4ec81010a8a24986e2ca8e512150eed15375db Mon Sep 17 00:00:00 2001 From: James Wheare Date: Wed, 22 Apr 2020 10:48:08 +0100 Subject: [PATCH 08/13] Clarify line break byte --- extensions/multiline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/extensions/multiline.md b/extensions/multiline.md index ad9858f51..a715c6796 100644 --- a/extensions/multiline.md +++ b/extensions/multiline.md @@ -69,7 +69,7 @@ When receiving a well-formed mulitiline message batch, implementations MUST coll * Servers: delivering the batch to the intended recipients * Clients: displaying the batched message to the user -Messages in a multiline batch MUST be concatenated with a line break unless the `draft/multiline-concat` message tag is sent, in which case the message is directly concatenated with the previous message without any separation. +Messages in a multiline batch MUST be concatenated with a single line feed (`\n`) byte unless the `draft/multiline-concat` message tag is sent, in which case the message is directly concatenated with the previous message without any separation. Clients MUST NOT send messages other than PRIVMSG while a multiline batch is open. From ff7f73bbb510e3c2caf504a84cb21fabebae9b48 Mon Sep 17 00:00:00 2001 From: James Wheare Date: Wed, 22 Apr 2020 10:49:08 +0100 Subject: [PATCH 09/13] Flesh out server considerations --- extensions/multiline.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/extensions/multiline.md b/extensions/multiline.md index a715c6796..3bf8b771e 100644 --- a/extensions/multiline.md +++ b/extensions/multiline.md @@ -157,7 +157,11 @@ Which would leave 353 bytes for each message. This section is non-normative. -In the context of flood protection, keep in mind that clients might send an entire batched multiline message all at once. +For server implementations and operators, consider the impact of relaying a large multiline batch. In particular, the potential to exhaust client send queue budgets, which will either result in client disconnection (a denial-of-service attack) or degraded client performance. + +Each additional protocol line carries extra metadata beyond the message content, such as the sender's full source mask, the command, and the target. This means that the byte size of a multiline batch sent to clients will typically be a function of max-lines as well as max-bytes. Configuring max-bytes, max-lines, and the send queue length together is reommended to mitigate these issues. + +Additionally, in the context of flood protection and throttling of incoming messages, keep in mind that clients might send an entire batched multiline message all at once. ## Examples From bacbc49558f6e452acf7a291cb0e386748607ed4 Mon Sep 17 00:00:00 2001 From: James Wheare Date: Fri, 24 Apr 2020 13:45:47 +0100 Subject: [PATCH 10/13] Review fixes --- extensions/multiline.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/extensions/multiline.md b/extensions/multiline.md index 3bf8b771e..4ae401f54 100644 --- a/extensions/multiline.md +++ b/extensions/multiline.md @@ -43,7 +43,7 @@ This specification adds the `draft/multiline` capability. Implementations requesting this capability indicate that they are capable of handling the batch type and message tag described below. -The capability has a REQUIRED value: a comma (`,`) (0x2C) separated list of tokens. Each token consists of a key which might have a value attached. If there is a value attached, the value is separated from the key by an equals sign (`=`) (0x3D). That is, `[=][,[=][,[=]]]`. Keys specified in this document MUST only occur at most once. +The capability MUST be advertised by servers with a REQUIRED value: a comma (`,`) (0x2C) separated list of tokens. Each token consists of a key with an optional value attached. If there is a value attached, the value is separated from the key by an equals sign (`=`) (0x3D). That is, `[=][,[=][,[=]]]`. Keys specified in this document MUST only occur at most once. Clients MUST ignore every token with a key that they don't understand. @@ -62,7 +62,7 @@ This specification adds the `draft/multiline` batch type, and introduces client In addition to the base batch parameters (reference-tag and type) a multiline batch has one additional parameter, the target recipient. -Multiline batches MUST contain one or more PRIVMSG lines whose target MUST match the batch target. +Multiline batches MUST only contain one or more PRIVMSG lines. These lines MUST all have a target which matches the batch target. When receiving a well-formed mulitiline message batch, implementations MUST collect each PRIVMSG message content and wait until the full batch has been received before processing the message. Processing in this context refers to: @@ -159,7 +159,7 @@ This section is non-normative. For server implementations and operators, consider the impact of relaying a large multiline batch. In particular, the potential to exhaust client send queue budgets, which will either result in client disconnection (a denial-of-service attack) or degraded client performance. -Each additional protocol line carries extra metadata beyond the message content, such as the sender's full source mask, the command, and the target. This means that the byte size of a multiline batch sent to clients will typically be a function of max-lines as well as max-bytes. Configuring max-bytes, max-lines, and the send queue length together is reommended to mitigate these issues. +Each additional protocol line carries extra metadata beyond the message content, such as the sender's full source mask, the command, and the target. This means that the byte size of a multiline batch sent to clients will typically be a function of max-lines as well as max-bytes. Configuring max-bytes, max-lines, and the send queue length together is recommended to mitigate these issues. Additionally, in the context of flood protection and throttling of incoming messages, keep in mind that clients might send an entire batched multiline message all at once. From 051b87a0fffb30ed13e3b158e9e8ee72cd048ebf Mon Sep 17 00:00:00 2001 From: James Wheare Date: Fri, 24 Apr 2020 13:51:30 +0100 Subject: [PATCH 11/13] Use the wording for batch dependency that we use elsewhere --- extensions/multiline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/extensions/multiline.md b/extensions/multiline.md index 4ae401f54..91edfb28a 100644 --- a/extensions/multiline.md +++ b/extensions/multiline.md @@ -35,7 +35,7 @@ Mulitline messages allow for a more coherent experience that avoids the disconne ### Dependencies -This specification uses the [batch][] capability which MUST be negotiated at the same time. It also uses the [message tags][] and [standard replies][] frameworks. +This specification uses the [`batch`][] capability which MUST be negotiated to use multiline messages. It also uses the [message tags][] and [standard replies][] frameworks. ### Capabilities From e8efcaecb96912df1d73b2f5b9d9b51b9a09f5ed Mon Sep 17 00:00:00 2001 From: James Wheare Date: Fri, 24 Apr 2020 14:02:30 +0100 Subject: [PATCH 12/13] Clients shouldn't split in the middle of multi-byte characters --- extensions/multiline.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/extensions/multiline.md b/extensions/multiline.md index 91edfb28a..49d1a5202 100644 --- a/extensions/multiline.md +++ b/extensions/multiline.md @@ -112,6 +112,8 @@ When using `draft/multiline-concat` to split longer lines, care is required to e The recommended splitting method is to split long lines between words where possible and leave the space character at the end of the line. This ensures that pairs of words spanning a line concatenation are still space separated, without adding a space to the start of any fallback lines. +Take care not to split in the middle of multi-byte character sequences. + In order to calculate the maximum allowed space before splitting a line, consider the following line: ``` From 63d5c9e19bbc65febb02a32ffa059e2089ce4084 Mon Sep 17 00:00:00 2001 From: James Wheare Date: Fri, 24 Apr 2020 14:24:11 +0100 Subject: [PATCH 13/13] Stronger wording to prevent negotiation order fussiness --- extensions/multiline.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/extensions/multiline.md b/extensions/multiline.md index 49d1a5202..9a2243f99 100644 --- a/extensions/multiline.md +++ b/extensions/multiline.md @@ -35,7 +35,9 @@ Mulitline messages allow for a more coherent experience that avoids the disconne ### Dependencies -This specification uses the [`batch`][] capability which MUST be negotiated to use multiline messages. It also uses the [message tags][] and [standard replies][] frameworks. +This specification depends on the [`batch`][] capability which MUST be negotiated to use multiline messages. The order of capability negotiation is not significant and MUST not be enforced. + +This specification also uses the [message tags][] and [standard replies][] frameworks. ### Capabilities