Support dynamic record sizes

[colmmacc]

s2n callers should be able to specify the maximum record size used for outbound TLS records. Some applications prefer large sizes (which optimize for throughput) and others prefer small sizes (for latency and memory conservation). These sizes should be changeable during the lifetime of a TLS session.

Alternatively, s2n could take responsibility for setting adaptive record sizes.

Additionally; for inbound record sizes it is useful to resize our buffers if the peer is sending small records, as this can conserve memory. Perhaps resize buffers to the min() of the last N records, where N ~= 5-10.

[venkat1109]

If we are to go with adaptive record sizing, that resizing decision has to be driven by tcp cwnd size ultimately. One idea would be to start small and then after transmitting x bytes of data, go big. Then downsize after y seconds of inactivity.

[mark-kubacki]

There's something similar for Nginx which you could use as template:

http://forum.nginx.org/read.php?29,258662,259191#msg-259191

https://github.com/wmark/ossdl-overlay/blob/576dbf36ff6a1d80a2fa4a025ee1c4ecf2c2e8cd/www-servers/nginx/files/nginx-1.9.1-SSL-dynamic-record-size-redux.patch

[igrigorik]

Related ATS bug & discussion: https://issues.apache.org/jira/browse/TS-2503

[venkat1109]

First pass at dynamic record sizing:

venkat1109@70498df
venkat1109@eac8c3d

There are a couple of things that I don't like about this change:

(1) Re-allocation / resizing of conn.out stuffer whenever the record size changes.
What do we do when re-allocation fails because of mlock() or madvise() ? Do we really want to bail the connection ? If so, how do we propagate the error to the caller ?

(2) Calls to s2n_timer_reset() inside of s2n_send()

Thoughts / suggestions ?

[fatrat1117]

Hi:
I am going to implement a simper logic to increase record size. here is my proposal:

Define:
outgoing_rec_threshold: the number of bytes to send before changing the record size.
outgoing_rec_timeout (in seconds). If this value > 0 then dynamic TLS record sizes feature is enabled. Otherwise, the feature is disabled (default).

Logic:
increase record size (phase 1):
Each connection starts with records of the min (S2N_SMALL_FRAGMENT_LENGTH, max_outgoing_fragment_length). if S2N_SMALL_FRAGMENT_LENGTH < max_outgoing_fragment_length, after sending outgoing_rec_threshold bytes the record size is increased to max_outgoing_fragment_length.

reset record size (phase 2):
if the connection idles for longer than outgoing_rec_timeout then the TLS record size is reduced to S2N_SMALL_FRAGMENT_LENGTH and the logic above is repeated.