The TLS (Transport Layer Security) record protocol is the core mechanism that ensures confidentiality and integrity of data as it travels between clients and servers. Its responsibilities include fragmenting application data into manageable records, applying optional compression, encrypting the data, and attaching a message authentication code (MAC) to defend against tampering. Each step in the process is crucial for maintaining the security guarantees that TLS provides.

When data is ready to be sent over a TLS connection, the record protocol first fragments the data into chunks that fit within the maximum record size. This fragmentation allows TLS to efficiently handle large streams of data, sending them in smaller, more manageable pieces. After fragmentation, the protocol may apply compression, although in modern TLS versions, compression is typically disabled to prevent certain attacks. Next, the protocol encrypts the data using symmetric encryption algorithms negotiated during the TLS handshake. Before encryption, a MAC is calculated over the data, providing a cryptographic fingerprint that allows the receiver to verify the data’s integrity and authenticity after decryption.

The following pseudo code outlines the process for encrypting and decrypting TLS records, highlighting the sequence of operations performed by the record protocol.

TLS Record Protocol: Encrypting and Decrypting Records

Encrypting a TLS Record:

1. Fragment application data into records;
2. Optionally compress the record;
3. Compute the MAC over the record data and header;
4. Append the MAC to the record;
5. Encrypt the record (data + MAC) using the negotiated cipher;
6. Send the encrypted record.

Decrypting a TLS Record:

1. Receive the encrypted record;
2. Decrypt the record using the negotiated cipher;
3. Separate the data and MAC;
4. Compute a new MAC over the decrypted data and header;
5. Compare the computed MAC with the received MAC;
6. If the MACs match, accept the data; otherwise, reject the record.

A message authentication code (MAC) is a cryptographic checksum computed over a message and a secret key, providing assurance that the message has not been altered in transit. In TLS, the MAC is generated using algorithms such as HMAC (Hash-based Message Authentication Code), which combines a hash function with a shared secret key. The MAC is appended to the record before encryption, so any tampering with the data or MAC after encryption will be detected when the record is decrypted and the MAC is verified.

The MAC serves two critical functions in TLS:

• Verifying data integrity: ensuring that the received data matches what was sent;
• Authenticating the sender: confirming that the data originated from a party possessing the shared secret key.

If an attacker modifies any part of the record in transit, the MAC verification will fail at the receiving end, and the record will be discarded. This mechanism is essential for preventing undetected data modification and replay attacks.

Pseudo Code for MAC Calculation and Verification in TLS

MAC Calculation (Sender):

mac = HMAC(secret_key, header || data)
record = data || mac

MAC Verification (Receiver):

mac_received = extract_mac(record)
data = extract_data(record)
mac_computed = HMAC(secret_key, header || data)
if mac_computed == mac_received:
    accept data
else:
    reject record