HTTP Explained: How Websites Really Work
HTTP (Hypertext Transfer Protocol) is the foundation of web communication. Web communication means data transfer between your browser and servers. In today's world, the ease with which we use websites is made possible by HTTP. HTTP is based on Request-Response Model. So, to understand the basics of HTTP or to understand how HTTP helps us in web communication, we need to understand Request-Response Model.
Request-Response Model:
Sometimes we also called it as client-server model. So, in this model client (browser) send a request to the server and server send the response (Ex- web page, images, videos) for that request.
Request:
In request, client (browser) generally asks for data for website like HTML file, CSS file, etc. Some of the common fields which are present in request are given below:
Methods: (e.g., GET, POST, PUT, DELETE) etc.
URL: (e.g., /home, /login)
Headers: (e.g., User-Agent, Content-Type)
Body: We use it when we need to send data for PUT or POST requests.
Now let’s understand with an example, suppose you typed www.xyz.com in the browser and press enter. Your browser sends a request Which will look somewhat like this.
Explanation:
GET /index.html: This HTTP method gives information that browser wants to retrieve a particular type of data (Ex – index.html, CSS file, etc.)
Host: It specifies the domain name for which the request is meant i.e. www.xyz.com
User-Agent: it gives the detail of operating system and browser used by the client.
Accept: it informs the server, the types of data which the client can process. Here, client prefers HTML or XML and some other formats like image.
Accept-Language: Tells the server the preferred language(s) for the response. In this case, English (US) is preferred.
Response:
The data in the response is what the client had requested. Some of the common fields in response are:
Status Code: (e.g., 200, 404, 500)
200 OK: Request successful.
404 Not Found: Requested page/resource not found.
500 Internal Server Error: Some issue from server side.
Headers: provide metadata about the response (e.g., content type, server, caching rules), it helps client understand how to handle data of response body.
Body: It contains the requested data like HTML, JSON etc.
Now let’s understand the response which we get from previous request. We will discuss some of the common fields in response:
Explanation:
Status code: This indicates the response status [200 OK ] which means the request was successful, and the requested data is sent back in the response body.
Headers:
Cache-Control: Instructs the browser to not store the response in cache. This ensures that the content is always fetched fresh.
Content-Type: Specifies the media type of the resource being sent. Here, it’s HTML with UTF-8 character encoding.
Content-Length: Specifies the size of the response body in bytes. Here, it tells the client that the body is 1234 bytes long.
Content-Encoding: Specifies that the response body is compressed using gzip to reduce the size of the response which improves loading speed.
Age: Indicates, how long the response has been cached (in seconds). In our case, the resource was cached for 3600 seconds (1 hour).
Body:
Body of an HTTP response contains the actual content sent from the server to the client. It includes various types of data, such as HTML, JSON, images, or files. It is actually processed by the client's browser.
Explanation of diagram:
This is a basic HTML document structure that displays a "Welcome" title in the browser tab and a "Hello, World!" message in the body of the page.
HTTP versions:
HTTP/1.1:
Before moving to the HTTP/1.1, lets talk about HTTP/1.0. There is a problem in HTTP/1.0 i.e. we have to establish a separate connection for every request-response. The client and server would establish a new connection, exchange the data, and then terminate the connection. Some of the problems are mentioned below:
Slow performance: Opening and closing TCP connections repeatedly introduces additional overhead due to the 3-way handshake.
High Resource Usage: Each new connection consumes resources on both the client and server, such as CPU and memory that’s why it is very inefficient. This problem was resolved by HTTP/1.1.
For this problem HTTP/1.1 introduced the concept of persistent connection, where a single TCP connection could handle multiple requests and responses. This reduced the need to repeatedly open and close connections. This improves performance in website loading and reduce resource consumption of both client and server.
Persistent connection:
In HTTP/1.1, the TCP connection stays open and can be used for multiple requests and responses until the client or server decides to close it. This is known as a Keep-Alive connection. This reduces the latency and making web pages load faster. It also reduces the overhead of creating a separate connection for every request. Now single connection can send multiple requests like such as downloading HTML, CSS, JavaScript files, and images.
Pipelining:
Http/1.1 had another feature called HTTP pipelining. Now the client can send multiple requests at once without even waiting for response. But there is a catch: response will come according to the processing sequence. This means if a request needs more computation, more time to process, all the others will have to wait, and it will block all the responses. This limitation was later fixed by HTTP/2 and HTTP/3.
HTTP/2:
HTTP/2 is an updated version of HTTP/1.1. It is designed to improve the web performance. Various features introduced, like multiplexing, header compression, server push and stream prioritization. Let’s discuss these features one by one:
Multiplexing:
The ability to send multiple requests and responses simultaneously over a single TCP connection. It allows multiple requests to be processed in parallel without waiting for each other. In multiplexing data is transfer with the help of streams.
Streams:
Streams are the various independent logical channel in a single TCP connection. Each stream has a unique identifier and can be processed independently. The delay of one stream does not impact the other streams. All resources can be loaded at the same time.
Real-life example:
Imagine a train with multiple compartments.
Train: A single TCP connection.
Compartments: Streams (each compartment represents a different stream).
Passengers: Frames (data packets that are part of each stream).
The train can carry all the compartments (streams) and passengers (frames) on the same track (connection) at the same time.
Frames:
In HTTP/2, the data of a stream is divided into small frames (packets). Unique stream ID is attached with the frames, The frames of each stream are interleaved (mixed) and sent together. The server separates the frames based on their stream ID and processes them and reconstructs the complete response or request.
Limitation of HTTP/2:
Despite multiplexing, if data loss occurs in one stream, first it retransmits the lost data because it is built on top of TCP. In TCP, when data is lost, it fetches the missing data from the sender first. Until the retransmission is complete, it blocks all streams, and this is called Head-of-Line (HOL) blocking. Later, HTTP/3 resolved this issue.
Header compression:
Header Compression in HTTP/2 is a mechanism designed to reduce the size of HTTP headers, improving performance by minimizing bandwidth usage and speeding up the data transfer. This is achieved using the HPACK compression algorithm which efficiently compresses the header.
Header fields: HTTP requests and responses include headers like User-Agent, Cookies, and Accept. These headers contain the repetitive information.
HPACK Algorithm: it provides a table that stores previously sent header fields during a connection, it allows us to reuse without sending the repetitive headers again and again. Next time if HTTP/2 wants to send the repetitive headers, it will send the reference to the already stored header in the table.
Binary Encoding: Headers are compressed into a binary format, and this further reduces the size of headers. HTTP/1.1 is a text-based protocol that exchanges data in human-readable formats (ASCII). HTTP/2 replaced this with a binary format, which processes data in a machine-readable binary format (0s and 1s). Binary format is more compact than text-based information.
- Faster Parsing: Machine (computer) understand binary data more efficiently and processes it faster than text data. It is also compatible to the hardware-level operations like memory access, arithmetic or logic operations, etc.
Server Push:
In http/1.1 if you want any data from server you have to request for each and every resource, ultimately this will take more time to load web pages. Server Push is a powerful feature of HTTP/2 that allows the server to send additional resources to the client without waiting for the client to request them, anticipating that the client may need them in the future.
Example- if browser make a request of index.html file but server also send the additional files like style.css and script.js
HTTP/2 Encryption:
HTTP/2 uses TLS (Transport Layer Security) for encryption. Encryption means securing the data to ensure that no unauthorized user can access or modify it.
HTTPS:
HTTPS (Hypertext Transfer Protocol Secure) is the secure version of HTTP; it uses SSL/TLS encryption to protect the client data from third party interception. It operates on port-443, whereas HTTP uses port-80 for communication.
HTTPS also authenticates the server using SSL/TLS certificates from trusted Certificate Authorities, verifying the website's identity.
It also helps in SEO optimization and improves your website ranking.
TLS (Transport Layer Security):
TLS is a cryptographic protocol used for data encryption and for secure communication. It encrypts data to prevent it from third party. It also detects the tempering of data during the transmission and it also verifies the server identity.
TLS Handshake:
The TCP handshake occurs before the TLS handshake. Without the TCP handshake, the TLS handshake cannot take place. So, first of all, a TCP connection is established between client and server for the reliable communication. Then, the TLS handshake takes place to enable data encryption and ensure secure data transmission. Now we will discuss about TLS handshake step-by-step:
ClientHello:
The browser (client) sends a ClientHello message to the server, which includes a list of supported encryption protocols and algorithms.ServerHello:
The server responds with a ServerHello message, which contains the chosen encryption algorithm and its TLS certificate.Certificate Verification:
The browser verifies the server's certificate by Checking it against a trusted Certificate Authority (CA) and if it is valid than handshake proceeds. [Certificate Authority (CA) – It’s a trusted organization which issues TLS certificate to the servers.]Key Exchange:
The client and server exchange encryption keys (e.g., using RSA or Diffie-Hellman methods) and a shared symmetric key is generated on both sides. if everything is valid and right then TLS connection is established successfully and now data is exchanged in the encrypted form.
RSA (Rivest-Shamir-Adleman):
Key exchange is the process by which a client and a server securely share or generate encryption keys to establish an encrypted communication channel. Here, we are only discussing RSA to give you an idea of how the encryption key works. Let’s discuss it step-by-step:
RSA is an Asymmetric Encryption Algorithm:
It is asymmetric encryption algorithm because it uses two keys: a public key (shared openly) and a private key (kept secret). Data encrypted with the public key and can only be decrypted with the corresponding private key.
The Server Sends Its Public Key to the Client:
The server generates a pair of keys: a public key and a private key.
The public key is sent to the client via an initial handshake. This key is safe to share because it cannot be used to decrypt data, only to encrypt it.
The private key remains securely stored on the server.
The Client Uses the Public Key to Encrypt a Secret Key:
The client generates a random secret key, also called a "pre-master secret".
The client encrypts the secret key with the help of the server's public key so that only the server can decrypt it by using private key.
This secret key will be used as the basis for creating the shared symmetric key for secure communication.
To ensure that even if someone intercepts the communication, they cannot see the secret key since it can only be decrypted with the private key.
The client transmits the encrypted secret key to the server over the network.
The Server Uses Its Private Key to Decrypt the Secret Key:
The server receives the encrypted secret key and uses its private key to decrypt it.
The server now has the same random secret key generated by the client.
Both the client and server now share the same secret key.
Now, this shared key is used for encryption, which is faster and more efficient for encrypting large amounts of data. RSA is used only for securely exchanging keys because it is slow and computationally expensive.
Stream prioritization:
Stream Prioritization is used to manage which requests should be processed first when multiple requests are sent over the same connection, to improve page load performance.
Each request (or stream) can be assigned a weight value between 1 and 256. The higher weight means the stream is more important, and thus it will be prioritized for transmission.
Assigning Priority:
Resources that are crucial for rendering the page, such as the main HTML file or CSS files, are assigned higher weights to ensure they are sent first like CSS file might get above 200.
Resources like images or non-essential JavaScript files, which are not immediately needed to render the page, might be assigned lower weights, such as 50 or 100.
If a parent stream is given higher priority, its dependent streams can also be treated with higher priority, ensuring they are transmitted sooner, this is called Parent-child dependency. Ex- If an image file might depend on HTML/CSS file than it will get higher priority.
HTTP/3:
HTTP/3 is a new version of web communication based on the QUIC (Quick UDP Internet Connections) protocol. It improves the web performance and security. It also overcomes the limitations of HTTP/2. Previous versions (HTTP/1.1 and HTTP/2) were based on TCP, but HTTP/3 uses UDP.
Features of HTTP/3 and QUIC:
Solving the TCP Connection Handshake Issue:
In HTTP/2, TCP required a 3-way handshake along with a separate TLS handshake, due to which extra latency occurred in the data transfer or web communication.
QUIC combines both the handshakes and instead of that a single handshake occurs which supports 0-RTT (Zero Round Trip Time) connections, helps to reduce latency to nearly zero during connection establishment.
[RTT- Round trip time (RTT) is the total time it takes for a signal to travel from the sender to the receiver and back again.]
Solving the Head-of-line blocking issue:
In HTTP/2, if a packet in one stream is lost, TCP blocks other streams until that packet is retransmitted. In QUIC, each stream is independent, so an issue with one stream does not impact the others.
let’s take a real-life example of four-lane highway, so a single UDP connection is equivalent to highway and four-lane act as streams. If there is a traffic jam in one lane, it will not block the traffic of another lane.
In QUIC, each stream has its own buffer for individual data processing but in HTTP/2, buffer of all streams is dependent on TCP layer.
TLS 1.3 (Built-in Encryption):
HTTP/3 by default uses TLS 1.3 encryption.
This means data transmission is secure without requiring an additional handshake, making web traffic fast and secure.
Faster Packet Recovery:
QUIC uses UDP, which normally doesn’t retransmit lost data or confirm if data is received. But QUIC adds its own rules for error detection and for recovery of lost data.
If packet is lost in any of the streams, it will not block the other streams. That specific lost packet(frames) is transmitted without effecting other streams.
Connection Migration:
In HTTP/2, when the network changes (e.g., switching from Wi-Fi to mobile data), the connection is broken, requiring a new connection to be established. This causes disruptions in communication. In contrast, HTTP/3, which uses QUIC, supports connection migration. the connection can seamlessly migrate without being interrupted, maintaining continuous communication.
This feature is especially beneficial for mobile devices and networks with unstable connections, as it reduces the risk of disruptions and improves reliability. Thus, HTTP/3 ensures a smoother experience across different networks.
UDP (User Datagram Protocol):
UDP is a transport layer protocol which is an alternative to TCP (Transmission Control Protocol). It is connectionless, means we don’t need a dedicated connection (like 3-way handshake in HTTP/2) to start the communication. UDP is fast but it lacks in reliability.
In UDP, sender send its data(packets) to the receiver’s specific IP Address along with the port number. It does not check the availability of the sender or receiver, nor does it confirm whether the packet was successfully received.
UDP focuses on speed and simplicity, that is why it skips complex mechanisms of reliability and error detection.
Features of UDP:
Connectionless: In UDP, no connection is established between the sender and receiver. This means the sender doesn’t know if the receiver is available before sending data.
No Acknowledgement: UDP doesn’t provide any acknowledgment after data transmission. If data is lost, the sender has no way of knowing.
No Retransmission: If a packet is lost, UDP doesn’t attempt to resend it. It tolerates data loss.
Low Latency: Since there’s no connection setup or acknowledgment process, UDP is ideal for low latency communication.
Faster: UDP is faster because it doesn’t have error correction or control mechanisms. It focuses solely on delivering data.
Use cases:
Real-Time Applications:
Voice Calls (VoIP): Where speed is more important, and small data losses can be tolerated.
Live Streaming: When continuous data flow is required, and losing some packets doesn't significantly affect the user experience.
Gaming:
- In online multiplayer games, the speed of real-time data exchange is important, which is why UDP is used. If some data packets are lost, it will not affect your gaming experience.
DNS (Domain Name System):
- UDP is used in DNS requests because our goal is to provide a fast and efficient response.
Video Conferencing:
- Applications like Zoom or Google Meet use UDP for real-time video and audio transmission, as these protocols help minimize latency.