Understanding HTTPS principles, SSL/TLS protocols in detail

Introduction to SSL/TLS

When we use the HTTP protocol, the data transmitted is insecure, because all data going to and from the client and the server is in plaintext:.

• Third parties can get access to the real data
• Third parties can tamper with the data
• Third parties can impersonate the server or client

The full name of HTTPS is Hypertext Transfer Protocol Secure, which is used to exchange information securely (secure communication) between two end systems on a computer network, which is equivalent to adding a Secure secure word eye to HTTP, then we can give a HTTPS definition: HTTPS is a protocol and specification for secure transmission of text, images, audio, video and other hypertext data between two points in the computer world. HTTPS is an extension of the HTTP protocol, which does not guarantee the security of the transmission itself, so who guarantees the security? In HTTPS, the communication protocol is encrypted using Transport Layer Security (TLS) or Secure Sockets Layer (SSL). That is, HTTP + SSL(TLS) = HTTPS.

SSL (Secure Sockets Layer) and TLS (Transport Layer Security) were designed by Netscape as secure transport protocols for the Web. The first versions of SSL (SSL 1.0, SSL 2.0, SSL 3.0) were designed and maintained by Netscape, and starting with version 3.1, the SSL protocol was officially taken over by the Internet Engineering Task Force (IETF) and renamed TLS. From version 3.1, the SSL protocol was officially taken over by the Internet Engineering Task Force (IETF) and renamed as TLS (Transport Layer Security), which has been developed into TLS 1.0, TLS 1.1, TLS 1.2 and TLS 1.3.

• In 1994, NetScape designed SSL 1.0
• In 1995, SSL 2.0, with serious vulnerabilities
• In 1996, SSL 3.0, which was adopted on a large scale
• In 1999, the IETF standardized on SSL and released TLS 1.0
• In 2006 and 2008, TLS was upgraded twice, to TLS 1.1 and TLS 1.2
• In 2018, TLS 1.3 was published as a recommended standard in RFC 8446

Since the 2 versions of SSL have been retired from history, only the name TLS is used later. The reader should understand that what is generally referred to as SSL is TLS.

SSL is a separate protocol that can be used not only by HTTP, but also by other application layer protocols such as SMTP (email protocol), Telnet (remote login protocol), etc.

The design goal of TLS

The design goal of TLS is to build a secure transport layer (Transport Layer Security) that provides, on top of a connection-based transport layer (such as tcp).

• cryptographic security
  • confidentiality, message privacy (confidentiality is achieved by encryption, where all messages are transmitted encrypted and cannot be eavesdropped on by third parties)
  • integrity, message integrity (through the MAC verification mechanism, once tampered with, both sides of the communication will immediately detect)
  • authentication, mutual authentication (both sides can be equipped with certificates to prevent identity impersonation)
• interoperability, universal (according to the public rfc, any rfc-compliant software implementation can interoperate, not limited by any proprietary technology)
• Extensibility (there are a large number of new features that can be added through the extension mechanism tls_ext)
• high efficiency (through session cache, after proper deployment of cache, tls is highly efficient)

How do I determine the version of TSL used on my website?

In your browser, press F12 to bring up the debug window and select the Security TAB to see something like the following.

Principle of TLS

Cryptographic communication protocol architecture: top-down, layered abstraction

A common way to build software is layered, abstracting the problem domain into multiple layers, with each layer’s concepts defined as a set of primitives that are implemented by the previous layer using the components of the next layer to construct and be used by the previous layer, and the layers are stacked to form software.

• For example, in the programming language domain, assembly language is a layer, above assembly is a static compiled language such as C/C++, above C/C++ is a dynamically typed scripting language layer such as python/php/lua, and above that often domain-specific DSLs are constructed
• In network architecture, Ethernet is a layer on top of which is the network layer of the ip protocol, above ip is the transport layer such as tcp, and above tcp is the application layer such as http

Cryptographic communication protocols are also constructed in layers to obtain. The layers can be roughly divided as follows.

• The bottom layer is the implementation of the basic algorithm language, such as: aes , rsa, md5, sha256, ecdsa, ecdh, etc.
• The top layer is the implementation of the standard cryptographic algorithms with selected parameters, including block encryption algorithms, signature algorithms, asymmetric encryption algorithms, MAC algorithms, etc., such as: aes-128-cbc-pkcs7, rsaes-oaep, rsassa-pkcs1-v1_5, hmac-sha256, ecdsa-p256, curve25519, etc.
• On top of that, it is a semi-finished component that combines multiple standard algorithms, for example: symmetric transport components such as aes-128-cbc + hmac-sha256, aes-128-gcm, authentication key negotiation algorithm: rsassa-OAEP + ecdh-secp256r1, digital envelope: rsaes-oaep + aes-cbc -128 + hmac-sha256, file password encryption storage component: pbkdf2 + aes-128-cbc-hmac-sha256, key extension algorithm PRF-sha256, etc.
• On top of that, there are various finished cryptographic protocols/software assembled with various components, such as: tls protocol, ssh protocol, srp protocol, gnupg file format, iMessage protocol, bitcoin protocol, etc.

For each layer.

• Layer 1, which is generally understood by programmers, such as rsa, is simply known to everyone; md5 is widely used (and, of course, widely misused)
• Layer 2, a variety of inexplicable parameters, generally very baffling to programmers, need to study in depth to sort out.
• Layer 3, many programmers build their own wheels, often to be frank, is just trying to repeat the implementation of a component of layer 3.
• Layer 4, properly understanding, using, and deploying such mature open protocols, is not so easy. A lot of misuse comes from a lack of understanding and requires a background knowledge of cryptography to figure out what, why, and how it works.

In layer 2, cryptographic algorithms, the following categories are common.

• block cipher: AES, Serpent, etc.
• Stream cipher: RC4, ChaCha20, etc.
• Hash functions: MD5, sha1, sha256, sha512 , ripemd 160, poly1305, etc.
• message authentication code: HMAC-sha256, AEAD, etc.
• key exchange: DH, ECDH, RSA, PFS method (DHE, ECDHE), etc.
• public-key encryption: RSA, rabin-williams, etc.
• signature algorithm: RSA, DSA, ECDSA (secp256r1 , ed25519), etc.
• key derivation function: TLS-12-PRF(SHA-256), bcrypto, scrypto, pbkdf2, etc.
• random number generators: /dev/urandom etc.

The process of designing a cryptographic communication protocol is a top-down process of selecting various components and assembling them into a complete protocol

TLS CipherSuite

From the above layered perspective, TLS is roughly spelled out in 3 components.

• Symmetric encrypted transport component, e.g. aes-128-gcm (these are examples of the most dominant choices currently available in 2015);
• an authentication key negotiation component, e.g. rsa-ecdhe;
• key extension components, e.g. TLS-PRF-sha256

These components can be subdivided into 5 classes of algorithms, which in TLS are combined together in a CipherSuite: the

• authentication (authentication algorithm)
• encryption (encryption algorithm)
• message authentication code (Message Authentication Code algorithm, MAC)
• key exchange (key exchange algorithm)
• key derivation function

The TLS protocol was designed with the evolution of each of these algorithms in mind, so instead of fixing the algorithm, an algorithm negotiation process was designed to allow the addition of new algorithms, and the negotiated combination of algorithms, a CipherSuite TLS CipherSuite, is registered in the iana set. for identification, which can be viewed at iana’s registration page

The list of supported CipherSuite on the server side, if openssl is used, can be viewed with the openssl ciphers -V | column -t command, the output is as follows

Architecture of the TLS protocol

The main purpose of TLS (Transport Layer Security Protocol) is to provide privacy and data integrity between two communicating applications. The protocol consists of two layers: the TLS Record protocol and the TLS Handshake protocol.

• The bottom layer is the TLS Record protocol, which is primarily responsible for encrypting messages using symmetric ciphers.
• The upper layer is the TLS Handshake protocol, which is divided into four parts: the handshake protocol, the cipher specification change protocol, and the application data protocol.
  • The handshake protocol is responsible for agreeing on the cryptographic algorithm and shared key on the client and server side, including certificate authentication, and is the most complex part of the 4 protocols.
  • The cryptographic specification change protocol is responsible for signaling a change in cryptographic method to the communicating object (somewhat redundant and has been removed in 3)
  • The warning protocol is responsible for communicating errors to the other party in the event of an error
  • The application data protocol is responsible for communicating the application data carried by TLS to the communicating object.

This structure of authenticated key negotiation + symmetric encrypted transport is common to most encrypted communication protocols.

TLS Handshake Protocol

Before communicating with SSL, the handshake protocol of SSL is first used to shake hands at both ends of the communication, negotiate the relevant security parameters to be used in the data transmission (e.g. encryption algorithm, shared key, materials required to generate the key, etc.), and to authenticate the identity of the other end.

The SSL build process has a total of 13 packets, with at least 9 packets required for the first build.

SSL establishes the first stage: the client first sends a ClientHello message to the server, the server receives the ClientHello message, and then sends a ServerHello message in response to the client.

ClientHello

The first step of the handshake is for the client to send a Client Hello message to the server, which contains a client-generated random number Random1, the encryption suite supported by the client (Support Ciphers), and the SSL Version, among other information.

The messages involved in ClientHello are specified as follows.

• Client Version: lists the protocol versions supported by the client by priority, preferring the latest protocol version that the client wishes to support.
• Client random numberRandom
• Session ID (Session id): If the client connects to the server for the first time, then this field will remain empty. The field is empty in the above image, indicating that this is the first time connecting to the server. If the field is not empty, it means that there was a previous connection to the server, during which time the server will use the Session ID to map the symmetric key and store the Session ID in the client browser, setting a time limit for the mapping. If the browser connects to the same server in the future (before the time limit expires), it will send the Session ID and the server will verify the mapped Session ID and use the previously used symmetric key to recover the Session, no full handshake is required in this case. Also called as SSL session recovery. It will be described later.
• Cipher suites: The client sends the server a list of cipher suites it already knows, this is prioritized by the client but it is entirely up to the server to decide whether to send them or not. there is a standard format for the cipher suites used in TLS. In the above message, the client sends 74 cipher suites. The server will select one of them to be the common cipher suite for both sides.
• Compression method: Compression is possible in order to reduce bandwidth. However, from the examples of successful attacks on TLS, where the attack when using compression can capture the parameters sent with HTTP hair, this attack can hijack cookies, a vulnerability we call CRIME. since TLS 1.3, the protocol has disabled TLS compression.
• Extension packets: Other parameters (such as server name, padding, supported signature algorithms, etc.) can be used as extensions.

These are part of the client greeting. If the client greeting has been received, the next step is the server acknowledgement and the server will send the server greeting.

ServerHello

After receiving the client hello the server must send a server hello message, the server checks the conditions of the client hello specifying things like TLS version and algorithm, if the server accepts and supports all the conditions it will send its certificate and other details, otherwise the server will send a handshake failure message.

If accepted, the second step is for the server to send a Server Hello message to the client, which determines a cipher suite from the Support Ciphers passed from the Client Hello, which determines which algorithms are used for subsequent encryption and digest generation, and also generates a random number, Random2. Note that both the client and server have two random numbers (Random1+ Random2), which will be used in the subsequent generation of the symmetric secret key.

The specific parameters involved in ServerHello.

• ServerVersion: The server will select the latest version supported by the client.
• ServerRandom: Both server and client will generate a 32-byte random number. This is used to create the encryption key.
• Encryption Suite: The server will select an encryption suite from the list of encryption suites sent by the client.
• Session ID: The server stores the agreed Session parameters in the TLS cache and generates the Session id corresponding to it. it is sent to the client along with the Server Hello. The client can write the agreed parameters to this Session id with a given expiration time. If the client connects to the server again before this expiration time, the server can check the cached parameters corresponding to the Session id and reuse them without a full handshake. This is very useful because both the server and the client can save a lot of computational costs. There are drawbacks to this approach when it comes to applications with huge traffic such as Amazon and Google. Millions of people connect to the server every day, and the server must keep a TLS cache of all Session parameters using Session keys. This is a huge overhead. To solve this problem, Session Tickets were added to the extension package, where the client can specify in the client hello whether it supports Session Tickets. the server will then create a new Session Ticket with a private key encrypted with only the server’s knowledge of the Session parameters. It will be stored on the client, so all Session data is stored only on the client computer, but the Ticket is still secure, because the key is known only to the server. This data can be included in the Client Hello as an extension named Session Ticket.
• Compression algorithm: if supported, the server will agree to the client’s preferred compression method.
• Extended Package

After this phase, the following is known to the client server.

• SSL version
• Key exchange, message authentication and encryption algorithms
• Compression method
• Two random numbers about the key generation.

SSL establishes the second phase: the server sends a message to the client.

The server initiates SSL handshake phase 2 and is the sole sender of all messages in this phase and the client is the sole recipient of all messages. This phase is divided into 4 steps.

• Certificate: The server sends the digital certificate and the entire chain to the root CA to the client, enabling the client to authenticate the server with the server public key in the server certificate.
• Server key exchange (optional): Here it depends on the key exchange algorithm
• Certificate request: The server may ask the client to authenticate itself.
• Server handshake completion: the end of phase 2 and the signal for the start of phase 3

Certificate message (optional) - certificate is required for the first establishment

This message is normally required to be included in the full flow of the SSL handshake, except that it is not required to be sent when the session resumes. The message contains an X.509 certificate, which contains the public key and is sent to the client to verify the signature or to encrypt the message during the key exchange.

This is the step where the server sends its certificate to the client to verify its identity, and the client verifies that it has passed and removes the public key from the certificate.

Server Key Exchange (optional)

Based on the CipherSuite information previously included in the ClientHello message, which determines the key exchange method (e.g. RSA or DH), the Server Key Exchange message contains a set of parameters required to complete the key exchange.

Because this is a DH algorithm, it is necessary to send the DH parameters used by the server. the RSA algorithm does not require this step.

In Diffie-Hellman, the client cannot compute the premaster key itself; both parties help to compute it, so the client needs to obtain the Diffie-Hellman public key from the server.

As can be seen from the above diagram, the key exchange is also protected by signatures at this point.

Certificate Request (optional) - can be a one-way authentication or a two-way authentication

This step is optional and may be used if there is a high demand for security in common. The server is used to authenticate the client. The server side sends a Certificate Request message, asking the client to send his own certificate for verification. The message contains the certificate types supported by the server side (RSA, DSA, ECDSA, etc.) and a list of all CAs trusted by the server side as certificate issuers, and the client will use this information to filter the certificates.

Server Hello Done

This message indicates that the server has sent all the information and is waiting for a message from the client next.

SSL establishes the third phase: the client receives a series of messages from the server and parses them, then sends the corresponding messages from this end to the server.

The client initiates SSL handshake phase 3 and is the sole sender of all messages in this phase and the server is the sole recipient of all messages. This phase is divided into 3 steps.

• Certificate (optional): In order to prove itself to the server, the client has to send a certificate message, this is optional and can be configured in IIS to force client certificate authentication.
• Client key exchange (Pre-master-secret): Here the client sends the prepared master key to the server, note that the server’s public key will be used here for encryption.
• Certificate verification (optional), sign the prepared secret and random number to prove the possession of (a) the public key of the certificate.

Certificate (optional)

If in the second phase the server-side asks for a client certificate, the client sends its own certificate in that phase. The server-side Certificate Request message sent earlier contains a list of certificate types and CAs supported by the server-side, so the client will choose the first certificate among its own that meets these two conditions and send it. If the client does not have a certificate, a no_certificate warning is sent.

Client Key exchange

If the client does not have a certificate, it sends a no_certificate warning.

If it is RSA algorithm, a 48-byte random number will be generated and then encrypted with the server’s public key and put into the message. If it is DH algorithm, this is the DH parameter sent to the client, and then the server and the client each calculate the same pre-master secret according to the DH algorithm.

This message is sent to the server using the server’s public key encryption. The server decrypts it with its own private key to get the pre-master key.(Prove to the server that it does hold the client certificate private key.)

Certificate verify (optional)

This message needs to be sent only if the client has sent its own certificate to the server side. It contains a signature that signs the HMAC value (with master_secret) of all handshake messages since the first message.

SSL Establishment Phase 4: The handshake protocol is completed and the SSL connection is established.

The client initiates SSL handshake phase 4, which brings the server to an end. This phase is divided into 4 steps, with the first 2 messages coming from the client and the last 2 messages coming from the server.

A secure connection is established, the client sends a Change Cipher Spec message and copies the negotiated CipherSuite into the current connection state. Then, the client sends a Finished message with the new algorithm, key parameters, which checks whether the key exchange and authentication process has been successful. It includes a checksum value that verifies the message for the entire handshake process of the client. The server also sends a Change Cipher Spec message and a Finished message. The handshake process is complete and the client and server can exchange application layer data for communication.

ChangeCipherSpec

Notification of a change in encoding, indicating that all subsequent messages will be sent with the mutually agreed encryption method and key (ChangeCipherSpec is a separate protocol, embodied in the packet as a single byte of data, used to inform the server that the client has switched to the state of the previously negotiated Cipher Suite and is ready to encrypt the data and transmit it using the previously negotiated Cipher Suite).

Clinet Finished

A client handshake completion notification, indicating that the client’s handshake phase has ended. This item is also the hash value of all the content sent earlier, and is used by the server for verification (the HMAC algorithm is used to calculate the digest of all the handshake messages received and sent, and then a pseudo-function PRF defined in RFC 5246 is used to The result is computed, encrypted and sent. (This data is to verify the encryption and decryption channel just established by the handshake before the official transmission of the application data.)

Server Finished

Notification of the end of the server-side handshake.

• Use the private key to decrypt the encrypted Pre-Master data, based on the two plaintext random numbers random_C and random_S exchanged before (Client Hello and Server Hello), and calculate the negotiation key:enc_key=Fuc(random_C, random_S, Pre-Master) ;
• Calculate the hash value of all previously received messages, then decrypt the encrypted_handshake_message sent by the client to verify the correctness of the data and key;
• Send a ChangeCipherSpec (informing the client that it has switched to the negotiated encryption suite state and is ready to encrypt the data using the encryption suite and Session Secret)
• The server also encrypts a Finish message to the client using Session Secret to verify the success of the encryption and decryption channel previously established by the handshake.

According to the previous handshake information, if both the client and the server can encrypt and decrypt the Finish message normally and the message is verified correctly, the handshake channel has been established successfully, and then both parties can use the Session Secret generated above to encrypt the data for transmission.

TLS logging protocol

The TLS logging protocol is responsible for message compression, encryption and data authentication.

The message is first segmented, then compressed, its message authentication code is calculated, and then encrypted using a symmetric cipher, which uses the CBC mode, the initial vector of which is generated by the master cipher. After the ciphertext is obtained, other information such as type, version and length are appended to make up the final message data.

The record protocol does symmetric encryption of application data and accounts for the majority of the traffic on a TLS connection.

Record protocol - accepts data from the application layer and does:

• fragmentation, and in the reverse direction, reassembly
• Sequence number generation, which generates a unique number for each data block to prevent it from being replayed or reordered
• Compression, an optional step that uses the compression algorithm negotiated by the handshake protocol to do the compression
• Encryption, using the key negotiated by the handshake protocol to do encryption/decryption
• Calculate HMAC, calculate HMAC on the data and verify the correctness of the HMAC of the received packet
• Send to tcp/ip, send data to TCP/IP for transmission (or other ipc mechanism).

SecurityParameters

The above processing at the record layer is based entirely on the following parameters in SecurityParameters.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

struct {
    ConnectionEnd          entity;
    PRFAlgorithm           prf_algorithm;
    BulkCipherAlgorithm    bulk_cipher_algorithm;
    CipherType             cipher_type;
    uint8                  enc_key_length;
    uint8                  block_length;
    uint8                  fixed_iv_length;
    uint8                  record_iv_length;
    MACAlgorithm           mac_algorithm;
    uint8                  mac_length;
    uint8                  mac_key_length;
    CompressionMethod      compression_algorithm;
    opaque                 master_secret[48];
    opaque                 client_random[32];
    opaque                 server_random[32];
} SecurityParameters;

The record layer uses the above SecurityParameters to generate the following 6 parameters (not all CipherSuite’s need all 6, if they don’t, they are empty):

1
2
3
4
5
6

client write MAC key
server write MAC key
client write encryption key
server write encryption key
client write IV
server write IV

When the handshake is complete and the above 6 parameters are generated, the connection state can be established. In addition to the above SecurityParameters, the connection state has the following parameters and updates the following parameters as data is sent/received.

• compression state : The state of the current compression algorithm.
• cipher state : the current state of the encryption algorithm, for block encryption algorithms such as aes, it contains the round key generated by cipher preprocessing (thanks to Dr. Wen for pointing it out) “round key”, and IV, etc.; for stream encryption, it contains the state information that allows the stream encryption to continue to encrypt and decrypt
• sequence number : each connection state contains a sequence number, and there are different sequence numbers for read and write states. sequence number must be set to sequence number when the connection starts transferring data. sequence number is of type uint64 and must not exceed 264-1. Sequence number must not be looped back. If a TLS implementation cannot avoid bypassing a sequence number, it MUST renegotiate. sequence number is incremented by 1 each time a record is sent. and the first Record transmitted MUST use 0 as the sequence number.

record layer segmentation

The data stream to be sent is first segmented, as shown above, into the following format.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

struct {
    uint8 major;
    uint8 minor;
} ProtocolVersion;

enum {
    change_cipher_spec(20), alert(21), handshake(22),
    application_data(23), (255)
} ContentType;

struct {
    ContentType type;
    ProtocolVersion version;
    uint16 length;
    opaque fragment[TLSPlaintext.length];
} TLSPlaintext;

• version field: defines the current negotiated TLS protocol version, for example, TLS 1.2 version is { 3, 3 }
• length field: that is, the length, tls protocol stipulates that the length must be less than 214, generally we do not want the length too long, because the decryption side needs to receive the whole record, in order to decrypt, the length is too long will lead to the decryption side need to wait for more rtt, increase the latency, and destroy the user experience, refer to the Web Performance Authority Guide TLS The long length will cause the decryptor to wait for more rtt, increase the latency and break the user experience.
• type field: to identify which of the four protocols is the current record.

record compression: TLS protocol defines optional compression, however, because compression led to the 2012 outbreak of CRIME attack, BREACH attack, so in actual deployment, be sure to disable compression.

Cryptographic protection at the record layer

The processed packet format is defined as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

struct {
    ContentType type;
    ProtocolVersion version;
    uint16 length;
    select (SecurityParameters.cipher_type) {
        case stream: GenericStreamCipher;
        case block:  GenericBlockCipher;
        case aead:   GenericAEADCipher;
    } fragment;
} TLSCiphertext;

The TLS protocol design goals of confidentiality (encryption), integrity (authentication) and anti-replay are implemented here. There are 3 types of implementations.

1. Block Cipher (CBC mode of operation) + HMAC: e.g. aes-128-cbc+hmac-sha256
2. Stream Cipher (RC4) + HMAC
3. Authenticated-Encryption using block cipher (GCM/CCM mode): e.g. aes-128-gcm

Block Cipher + HMAC and Stream Cipher + HMAC of all kinds of algorithms have now (2015) have exploded various vulnerabilities, the most reliable is currently 3.Authenticated-Encryption class of algorithms, mainly aes-gcm, the next generation of TLS v1.3 only retains Authenticated-Encryption, the 1 and 2 directly prohibited.

GCM mode is AEAD, so it does not need MAC algorithm. GCM mode is a type of AEAD, and AEAD works similar to Encrypt-then-HMAC, e.g. Sha256 + Salt + AES + IV

A pitfall needs to be introduced here. In the history of cryptography, three combinations of encryption and authentication have emerged.

1. Encrypt-and-MAC
2. MAC-then-Encrypt
3. Encrypt-then-MAC

In the era when the TLS protocol was first defined, people did not realize the difference in security between these 3 combinations, so the TLS protocol specified the use of 2. MAC-then-Encrypt, i.e., first calculate the MAC, and then encrypt the “plaintext+MAC” (block encryption or stream encryption) to do stream encryption+MAC, and block encryption+MAC. However, tragically, in recent years, it has been found that this structure of MAC-then-Encrypt leads to easy construction of padding oracle-related attacks, such as this in TLS, which indirectly forms exploited by attackers, which indirectly leads to the BEAST attack , Lucky 13 attack (CVE- 2013-0169), and the POODLE attack (CVE-2014-3566).

As a result, the academic community has now agreed that Encrypt-then-MAC is the most secure! In view of this trap is so sinister, some people in the academic community have proposed, simply set Encrypt and MAC directly into an algorithm, solve the security problem inside the algorithm, no longer let the coders choose, to avoid the crowd of coders again by this trap pitted, which is AEAD (Authenticated-Encryption With Additional data ) class of algorithms, GCM mode is the most important kind of AEAD.

Cryptographic protection of the record layer - MAC

TLS record layer MAC calculation method.

1
2
3
4
5

MAC(MAC_write_key, seq_num +
                      TLSCompressed.type +
                      TLSCompressed.version +
                      TLSCompressed.length +
                      TLSCompressed.fragment);

where seq_num is the sequence number of the current record, each record will be ++, you can see the seq_num, and record header inside several fields are also counted, so as to solve the problem of anti-playback, and to ensure that any field of the record can not be tampered with.

After calculating the MAC, the format is as follows.

1
2
3
4

stream-ciphered struct {
    opaque content[TLSCompressed.length];
    opaque MAC[SecurityParameters.mac_length];
} GenericStreamCipher;

Then, according to SecurityParameters.cipher_type, choose the corresponding symmetric encryption algorithm to encrypt.

record layer cryptographic protection - stream cipher

stream cipher: counting stream cipher, the state of stream cipher is multiplexed between successive records. The main power of stream cipher is RC4, but currently RC4 has exploded multiple vulnerabilities, so in practice, the stream cipher algorithm is basically not used.

record layer cryptographic protection - CBC block cipher

CBC mode block cipher TLS currently has few reliable block cipher, basically AES (the most reliable, the most mainstream), Camellia, SEED, (3DES, IDEA and so on already seems old, DES please disable), the encryption finished format is as follows.

1
2
3
4
5
6
7
8
9

struct {
    opaque IV[SecurityParameters.record_iv_length];
    block-ciphered struct {
        opaque content[TLSCompressed.length];
        opaque MAC[SecurityParameters.mac_length];
        uint8 padding[GenericBlockCipher.padding_length];
        uint8 padding_length;
    };
} GenericBlockCipher;

This one is worth talking about, because we programmers usually see a lot of AES-CBC in the industry, with several parameters.

• IV: requires that it must be generated with a cryptographically secure pseudo-random number generator (CSPRNG) and must be unpredictable, which under Linux means using /dev/urandom, or using RAND_bytes() from the openssl library. Note: TLS did not have this IV field before version 1, and the last block of the previous record was used as the IV of the next record, and then the coarse thing was done, which led to the BEAST attack. So, TLS1.2 was changed to this. (For those of you who are still using CBC, we recommend paying attention to how your IV fields are generated. If you want to use it, do a good job of keeping it consistent with what TLS1.2 does). Where SecurityParameters.record_iv_length must be equal to SecurityParameters.block_size. For example, the IV of AES-256-CBC must be 16 bytes long, because the block size of AES 128/192/256 is 16 bytes.
• padding: use the common CBC PKCS 7 padding (in the block size = 16 bytes in this case, and PKCS 5 algorithm is the same thing, java code can be so used in this case, and the results of PKCS 5 is the same).
• padding_length: is the last byte of PKCS 7 padding. Note the 2 sinister traps.
  • the implemented code must receive all plaintexts before transmitting the ciphertext, otherwise there may be a BEAST attack
  • The implementation may perform a time-side channel attack based on the time calculated by the MAC, so it must be ensured that the -running time is irrelevant to whether the padding is correct or not.

record layer of cryptographic protection - AEAD cipher

AEAD to our focus on AEAD, AEAD is the emerging mainstream encryption mode, is currently the most important mode, where the mainstream AEAD mode is aes-gcm-128/aes-gcm-256/chacha20-poly1305

The finished AEAD encryption format is.

1
2
3
4
5
6

struct {
   opaque nonce_explicit[SecurityParameters.record_iv_length];
   aead-ciphered struct {
       opaque content[TLSCompressed.length];
   };
} GenericAEADCipher;

The inputs for AEAD ciphers are: key, nonce, plaintext, and “additional data”. The key is either client_write_key or the server_write_key. The MAC key is not required.

Each AEAD algorithm has to specify a different nonce construction algorithm and the length of GenericAEADCipher.nonce_explicit. In TLS 1.2, it is specified that in many cases, the technique of rfc5116 section 3.2.1 can be followed. where record_iv_length is the length of the explicit part of the nonce, and the implicit part of the nonce is derived from the key_block as client_write_iv and and server_write_iv, and the explicit part is placed in GenericAEAEDCipher.nonce_explicit. explicit.

In TLS 1.3 draft, this was changed:

• Specify that the length of the AEAD algorithm nonce is specified as max(8 bytes, N_MIN), i.e., if N_MIN is larger than 8, use N_MIN; if it is smaller than 8, use 8.
• Specify that AEADs with N_MAX less than 8 bytes shall not be used for TLS.
• Specify that the nonce of each record in a TLS AEAD is constructed by filling the right side of the 64-bit sequence number with zeros until the length reaches iv_length. write_iv (depending on the sender’s choice). After the completion of the XOR, the obtained nonce of iv_length can be used as the nonce of each record.

The plaintext of AEAD input is TLSCompressed.fragment (remember the introduction above? AEAD is the integration of MAC and encrypt, so the input data does not need to count MAC).

AEAD input additional_data is:

1
2

additional_data = seq_num + TLSCompressed.type +
                  TLSCompressed.version + TLSCompressed.length;

The “+” indicates string splicing. You can see that here, similar to the MAC calculation above, the seq_num is counted to prevent replay and the type,version,length and other fields prevent these metadata from being tampered with.

1
2

AEADEncrypted = AEAD-Encrypt(write_key, nonce, plaintext,
                             additional_data)

Decryption + Verification of Integrity.

1
2
3

TLSCompressed.fragment = AEAD-Decrypt(write_key, nonce,
                                            AEADEncrypted,
                                            additional_data)

If the decryption/verification integrity fails, a fatal bad_record_mac alert message will be replied.

The iv length, nonce length, nonce composition, etc. of aes-gcm will be discussed in depth later.

Cryptographic protection of record layer - Key extensions

The master_secret generated by the TLS handshake is only 48 bytes, and the length of the two encryption keys, MAC key, and IV are generally longer than 48 (e.g. AES_256_CBC_SHA256 requires 128 bytes), so a function is used inside TLS to extend the 48 bytes to the required length, called PRF.

1
2
3
4

key_block = PRF(SecurityParameters.master_secret,
                "key expansion",
                SecurityParameters.server_random +
                SecurityParameters.client_random);

Then, the key_block is split like the following.

1
2
3
4
5
6

client_write_MAC_key[SecurityParameters.mac_key_length]
server_write_MAC_key[SecurityParameters.mac_key_length]
client_write_key[SecurityParameters.enc_key_length]
server_write_key[SecurityParameters.enc_key_length]
client_write_IV[SecurityParameters.fixed_iv_length]
server_write_IV[SecurityParameters.fixed_iv_length]

TLS uses the HMAC structure, and the hash function specified in CipherSuite (with a security level of at least SHA256) to construct the PRF.

First define P_hash, which extends (secret,seed) into an infinitely long stream of bytes.

1
2
3

P_hash(secret, seed) = HMAC_hash(secret, A(1) + seed) +
                       HMAC_hash(secret, A(2) + seed) +
                       HMAC_hash(secret, A(3) + seed) + ...

where “+” indicates string splicing. A() is defined as :

1
2

A(0) = seed
A(i) = HMAC_hash(secret, A(i-1))

The PRF of TLS is to apply the P_hash to the secret:

1

PRF(secret, label, seed) = P_<hash>(secret, label + seed)

where label is a protocol-defined, fixed ASCII string.

Note that this approach has been deprecated in TLS 1.3 in favor of the more reliable HKDF, which is also one of the standard algorithms for html5’s WebCryptoAPI.

Digital certificates

Digital certificates, also known as “identity cards” on the Internet, are used to uniquely identify an organization or a server, which is similar to the “resident ID” we use in our daily lives to uniquely identify a person.

The same is true for website certificates. Generally speaking, digital certificates are bought from a trusted authority (Certification Authority). Generally browsers are shipped with many well-known CAs built in.

In practice, HTTPS does not directly transmit the public key information, but uses the digital certificate carrying the public key information to ensure the security and integrity of the public key. It is used to protect the public key information sent to the client by the HTTPS server from being tampered with.

Digital Certificates and CA Authorities

A digital certificate typically contains.

• Server public key
• Holder information
• Certificate Authority (CA) information
• The CA’s digital signature for this document and the algorithm used
• The certificate expiration date
• And some other additional information

The purpose of a digital certificate is to authenticate the identity of the public key holder to prevent impersonation by third parties. To put it simply, the certificate is used to tell the client whether the server is legitimate, because only if the certificate is legitimate, it means the identity of the server is trusted.

We use a certificate to authenticate the identity of the public key holder (the identity of the server), but where does the certificate come from? And how should we certify the certificate? In order to make the public key of the server side trusted, the certificates of the server side are signed by CA (Certificate Authority), which is the public security bureau and notary center in the network world and has high credibility, so it signs each public key and the certificate issued by the trusted party is inevitably trusted. The reason why the signature is needed is because the role of the signature can avoid the tampering of the certificate content by the intermediary in obtaining the certificate.

Digital certificate issuance and verification process

The following figure shows the digital certificate issuance and verification process.

• The process of CA issuing certificate, as shown in the left part of the above figure.
  • First the CA will type the holder’s public key, usage, issuer, valid time and other information into a package (blue), and then perform Hash calculation on this information to get a Hash value
  • Then the CA will encrypt the Hash value with its own private key to generate Certificate Signature, that is, the CA has signed the certificate
  • Finally, the Certificate Signature is added to the file certificate to form a digital certificate.
• The client verifies the digital certificate of the server, as shown in the right part of the above figure.
  • First the client will use the same Hash algorithm to calculate the Hash value of the certificate H1
  • The browser receives the certificate from the server and decrypts the Certificate Signature content using the CA’s public key to obtain a Hash value H2
  • Finally, compare H1 and H2, if the value is the same, the certificate is trusted, otherwise the server certificate is considered untrustworthy

Certificate Chain

But in fact, there is a certificate trust chain problem in the certificate verification process, because the certificate we apply to the CA is generally not issued by the root certificate, but by the intermediate certificate, such as Baidu’s certificate, you can see from the following figure, the certificate hierarchy has three levels.

The verification process for this certificate with three-level hierarchical relationship is as follows.

• After the client receives the certificate of com, it finds that the issuer of this certificate is not the root certificate, so it cannot verify whether the baidu.com certificate is trusted according to the public key in the locally available root certificate. So, the client finds that the issuer of this certificate is “GlobalSign Organization Validation CA - SHA256 - G2” according to the issuer in the baidu.com certificate and then requests this intermediate certificate from the CA.
• After requesting the certificate, it finds that the certificate of “GlobalSign Organization Validation CA - SHA256 - G2” is issued by “GlobalSign Since “GlobalSign Root CA” has no higher level issuing authority, it is a root certificate, that is, a self-signed certificate. The application software will check whether this certificate has been preloaded on the root certificate list, and if so, it can use the public key in the root certificate to verify the “GlobalSign Organization Validation CA - SHA256 - G2 " certificate, and if the validation is found to be passed, the intermediate certificate is considered to be trusted.
• After the “GlobalSign Organization Validation CA - SHA256 - G2” certificate is trusted, you can use the “GlobalSign Organization Validation CA - SHA256 - G2” certificate to verify the trustworthiness of the com certificate, and if the verification is passed, the baidu.com certificate can be trusted.

In these steps, at first the client only trusts the root certificate GlobalSign Root CA certificate, then the “GlobalSign Root CA” certificate trusts the “GlobalSign Organization Validation CA - SHA256 - G2” certificate, and “GlobalSign Organization Validation CA - SHA256 - G2” certificate in turn trusts baidu.com certificate, so the client also trusts baidu.com certificate.

In summary, since the user trusts GlobalSign, baidu.com guaranteed by GlobalSign can be trusted, and since the user trusts the software vendor of the operating system or browser, GlobalSign with the root certificate preloaded by the software vendor can be trusted.

There are usually some root certificates built into the operating system.

Such a layer of verification constitutes a trust chain, and the entire certificate trust chain verification process is shown in the following figure.

The last question is, why do we need such a troublesome process of certificate chain, and why Root CA does not issue certificates directly, but has to make so many intermediate levels?

This is to ensure the absolute security of the root certificate, the stricter the isolation of the root certificate the better, otherwise if the root certificate is lost, then the whole trust chain will have problems.

TLS 1.3 vs TLS 1.2

TLS 1.3 is a new update to previous versions such as TLS 1.2 after nine years, and is the most significant change to date. In response to known security threats, the IETF (Internet Engineering Task Force) is working on a new standard for TLS 1.3 that promises to be the most secure, but also the most complex TLS protocol ever.

TLS 1.3 differs significantly from previous versions of the protocol, mainly in that

• Introduction of a new key negotiation mechanism - PSK - compared to previous versions
• Support for 0-RTT data transfer, which saves round-trip time when establishing a connection
• Deprecated cryptographic components such as 3DES, RC4, AES-CBC, and hash algorithms such as SHA1 and MD5
• All handshake messages after ServerHello are encrypted, and the visible plaintext is greatly reduced.
• Compression of encrypted messages and renegotiation initiated by both parties are no longer allowed.
• DSA certificates are no longer allowed in TLS 1.3

TLS 1.3 is indeed a big step forward compared to the shortcomings of the old protocol. It avoids the flaws of the previous version and reduces the time required for the TLS handshake.

To summarize, TLS 1.3 has two big advantages over previous versions, which are.

Faster access speed

To compare the changes in TLS 1.3 during the TLS handshake phase, here is a comparison of TLS 1.2 and TLS 1.3 during the TLS handshake phase.

As you can see from the above diagram, using TLS 1.2 requires two round trips (2-RTT) to complete the handshake before the request can be sent.

The TLS 1.3 handshake no longer supports static RSA key exchange, which means that a full handshake must be performed using Diffie-Hellman with forward security. As you can see from the figure above, only one round trip (1-RTT) is required to complete the handshake using the TLS 1.3 protocol.

The right part of the diagram above is the handshake process of TLS 1.3. You can see that TLS 1.3 merges the two messages Hello and Public Key Exchange into one message, so that it reduces to only 1 RTT to complete the TLS handshake.

How is this done? The client brings the supported elliptic curves in the Client Hello message, along with the public keys corresponding to those elliptic curves.

The server receives it, selects an elliptic curve and other parameters, and then returns the message with the server’s side of the public key. After this 1 RTT, both sides already have the material to generate the session key in hand, so the client calculates the session key and can carry out the encrypted transmission of application data.

Compared to TLS 1.2, TLS 1.3 handshake time is reduced by half. This means that accessing a mobile site, using the TLS 1.3 protocol, may reduce the time by almost 100ms.

Stronger Security

TLS has been in development for over 20 years. In previous versions, TLS 1.2 was highly configurable for better compatibility with older versions of browsers, which meant that vulnerable sites were always running insecure encryption algorithms that allowed Internet hackers to take advantage of them.

TLS 1.3 builds on previous versions by removing those insecure encryption algorithms, which include

• RSA key transfer - does not support forward security
• CBC mode ciphers – vulnerable to BEAST and Lucky 13 attacks
• RC4 stream ciphers – not secure for use in HTTPS
• SHA-1 hash function - recommended to be replaced by SHA-2
• Arbitrary Diffie-Hellman group – CVE-2016-0701 vulnerability
• Exported passwords - vulnerable to FREAK and LogJam attacks

TLS1.3 “slims down” the cipher suite by deprecating RSA and DH algorithms for key exchange algorithms, which do not support forward security, and supporting only ECDHE algorithms. For symmetric encryption and signature algorithms, only the most secure cipher suites are supported, for example, only the following 5 cipher suites are supported in openssl.

• TLS_AES_256_GCM_SHA384
• TLS_CHACHA20_POLY1305_SHA256
• TLS_AES_128_GCM_SHA256
• TLS_AES_128_CCM_8_SHA256
• TLS_AES_128_CCM_SHA256

The reason why TLS1.3 only supports so few cipher suites is that TLS1.2 supports various old and insecure cipher suites, so a man-in-the-middle can use a degradation attack to forge the Client Hello message from the client and replace the client’s supported cipher suite with some insecure cipher suite, so that the server is forced to use this cipher suite for HTTPS connection, thereby cracking the cipher text. The server is then forced to use this cipher suite for HTTPS connection, thus cracking the ciphertext.

In short, TLS 1.3 will be more secure than older versions of the TLS protocol, which also represents a major advance in Internet security.

TLS 1.3 specific changes

Changes to TLS 1.3 Significant improvements of note include

• RTT support
• RTT handshake support
• Change to use HKDF for key expansion
• Complete disabling of RC4
• Complete prohibition of compression
• Completely disable algorithms other than aead
• Remove explicit IV from aead
• Remove the length field in the AD of aead
• Remove ChangeCipherSpec
• Remove renegotiation
• Removal of static RSA and DH key negotiation

The improvements in TLS 1.3 are mainly for mobile Internet scenarios. TLS 1.3 removes ChangeCipherSpec so that there are three protocols on top of record: handshake, alert, and application data

Record layer cryptography protection changes

Since only aead is retained, so there is no need for MAC key. aead specific parameter usage has also been adjusted, the previous article has. kdf replaced with standard HKDF, there are 2 types of tls_kdf_sha256, tls_kdf_sha384

handshake protocol changes

In view of the session ticket is so good, simply people love, so TLS 1.3 directly built the session ticket, and renamed PSK. to note that this PSK and tls 1.2 in a very remote psk (see rfc4279 ) is not the same thing.

After considering session resuming, session ticket, TLS 1.3 proposes three handshake models.

• Diffie-Hellman (including both DH and ECDH, please make up your own mind to “ECDH/DH” where ECDH is mentioned below).
• A pre-shared symmetric key (PSK), a pre-shared symmetric key, here with a unified model for session resuming and rfc4279’s psk
• A combination of a symmetric key and Diffie-Hellman , the first two together

1-RTT Handshake

First, the TLS 1.2 handshake has 2 rtt, the first rtt is ClientHello/ServerHello, the second rtt is ClientKeyExchange/ServerKeyExchange, the reason why KeyExchange is placed in the second rtt is because tls1.2 has to support A variety of key exchange algorithms, and a variety of different parameters (such as DH or ECDH or RSA, ECDHE with what curve, DH with what group generation element, with what modulus, and so on), these algorithms and parameters rely on the first rtt to negotiate out, TLS1.3 boldly cut a variety of custom DH group, cut the ECDH custom curve, cut the RSA In fact, we basically use ECDH P-256 in practical applications, and no one uses anything else, so we simply let the client cache what negotiation algorithm the server used last time, and put KeyExchange directly into the first rtt, and the client directly uses the cached algorithm to send the public key of KeyExchange in the first rtt. If the server finds that the algorithm sent up by the client is not correct, then it will tell the correct one and let the client retry. In this way, the HelloRetryRequest message is introduced.

This brings it down to 1-RTT with basically no side effects. This is the full handshake for TLS 1.3.

Obviously, if a protocol has only one key negotiation algorithm, for example, set dead to ECDH P-256, it must be possible to do 1-RTT

The 0-RTT handshake with side effects

0-RTT should be inspired by Google’s QUIC protocol, if the server caches its ECDH public key in the client for a long time, then the client can use the ECDHE public key in the cache to construct an electronic envelope, in the first RTT, and send the application layer data directly. This long-term cached ECDH public key in the client is called semi-static (EC)DH share ECDH public key is sent to the client via ServerConfiguration message.

This 0-rtt optimization has the following side effects.

• Application data sent by RTT has no forward security.
• Application data in 0-RTT can be replayed across connections (any server-side protocol without shared state cannot be replay-proof across connections)
• If the private key corresponding to the server-side semi-static ECDH public key is leaked, the attacker can disguise as a client to tamper with the data at will.

The server sends the semi-static ECDH public key to the client in the ServerConfiguration message. ServerConfiguration is worth looking at.

1
2
3
4
5
6
7
8

struct {
    opaque configuration_id<1..2^16-1>;
    uint32 expiration_date;
    NamedGroup group;
    opaque server_key<1..2^16-1>;
    EarlyDataType early_data_type;
    ConfigurationExtension extensions<0..2^16-1>;
} ServerConfiguration;

where expiration_date is the last expiration date of this ServerConfiguration. This value is never allowed to be greater than 7 days. Clients are never allowed to store ServerConfiguration for more than 7 days, no matter how the server fills in this value.

The application data in 0-RTT is sent in EarlyDataIndication, and TLS 1.3 also defines a ContentType for EarlyDataIndication: early_handshake (four types of alert(21), handshake(22), application_data(23), early_handshake(25))

Resumption and PSK

In TLS 1.3, the key recovered from session resumption/session ticket, and psk (rfc4279) are unified in a handshake PSK mode.

PSK CipherSuite can use PSK and ECDHE together, so that there is forward security. It is also possible to use PSK only, so there is no forward security.

Changes to the Key Schedule process

In TLS 1.3, after considering the various cases of session tickets, the concepts of ES and SS were introduced to unify the various cases of key negotiation. In various handshake modes, ES and SS take different values from different sources.

• Ephemeral Secret (ES): the value derived from fresh ECDHE negotiation for each connection. Any value derived from ES is forward-secure (not forward-secure in PSK only mode, of course).
• Static Secret (SS): a value derived from a static, or semi-static key. For example, psk, or the server’s semi-static ECDH public key.

In various handshake modes.

As in the table above.

• For a full 1-RTT handshake, the ephemeral key is used for both SS and ES, which is certain to have forward security.
• With 0-RTT handshake, SS is generated using the ephemeral key on the client side and the semi-static ECDH public key on the server side.
• Pure PSK, this scenario has no forward security at all and should be avoided.
• PSK + ECDHE, this scenario is more interesting, SS is using Pre-Shared Key, which has no forward security, and ES is using ephemeral key, which has forward security.

As you can see, compared with the session ticket of TLS 1.2, the PSK + ECDHE of TLS 1.3 is combined with ES, which has forward security and is relatively more secure.

Unlike TLS 1.2, the master_secret of TLS 1.3 is derived using both ES and SS.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

HKDF-Expand-Label(Secret, Label, HashValue, Length) = HKDF-Expand(Secret, Label + '<pre class="EnlighterJSRAW" data-enlighter-language="generic">HKDF-Expand-Label(Secret, Label, HashValue, Length) = HKDF-Expand(Secret, Label + '\0' + HashValue, Length)
1. xSS = HKDF(0, SS, "extractedSS", L)
2. xES = HKDF(0, ES, "extractedES", L)
3. master_secret = HKDF(xSS, xES, "master secret", L)
4. finished_secret = HKDF-Expand-Label(xSS, "finished secret", handshake_hash, L)
</pre>' + HashValue, Length)
1. xSS = HKDF(0, SS, "extractedSS", L)
2. xES = HKDF(0, ES, "extractedES", L)
3. master_secret = HKDF(xSS, xES, "master secret", L)
4. finished_secret = HKDF-Expand-Label(xSS, "finished secret", handshake_hash, L)

Traffic Key Calculation

The key used for encrypting traffic is called Traffic Key in TLS 1.3. Due to the introduction of one more ContentType, the Traffic Key is not the same under different ContentTypes. The following table.

The concern is that the Traffic Key for Early Data is calculated using xSS. That is, it is determined using the Pre-Shared Key. Therefore, there is no forward security.

It is up to CipherSuite to negotiate which handshake mode to use for a TLS connection.