Reading an HTTP Request

Read a complete request from the socket and split it into a method, a target, a version and headers.

In the previous page, we built a TCP server that can accept connections from clients. But accepting a connection is only the beginning. When a browser connects to your server, it sends data describing what it wants. For example, a request to fetch a web page. Our TCP server currently sees only a stream of bytes. It doesn't know whether the client is asking for /, /about, or /images/logo.png.

To understand these requests, we need to teach our server how to read an HTTP request and it's basic structure.

We watched real requests arrive at our server, but as a single lump of bytes, and we only read them with one recv(1024) call. That leaves us with two problems:

One recv() is not guaranteed to return the whole request.
Even a complete request is just text. Our program has no idea which part is the method, which part is the path and which parts are headers.

Read The Complete Headers

TCP is a stream of bytes, and recv() returns whatever has arrived so far, up to the number we ask for. A request can show up in one piece, or in several. HTTP gives us a way to know when it is finished: the blank line that ends the headers, which is the four bytes \r\n\r\n.

So the rule is simple: keep reading, keep what we have, and stop when we see \r\n\r\n Let's put that in a function.

Add read_request() above main(), and use it in place of the single recv() call:

server.py

import socket

HOST = "127.0.0.1"
PORT = 8000

def read_request(conn: socket.socket) -> bytes:
    data = b""
    while b"\r\n\r\n" not in data:
        chunk = conn.recv(1024)
        if not chunk:  # the client closed the connection
            break
        data += chunk
    return data

def main() -> None:
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as server:
        server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        server.bind((HOST, PORT))
        server.listen()
        print(f"listening on http://{HOST}:{PORT} (Press Ctrl+c to stop)")
        
        try:
            while True:
                conn, addr = server.accept()
                with conn:
                    print(f"connection from {addr}")
                    data = read_request(conn)
                    print(data)
        except KeyboardInterrupt:
            print("\nbye")

if __name__ == "__main__":
    main()

Let's walk through the new read_request() function:

We start with an empty bytes buffer, data.
while b"\r\n\r\n" not in data keeps looping until the blank line appears anywhere in the buffer. Because we check the whole buffer and not just the newest chunk, it even works when these four bytes arrive and split across two recv() calls.
Each recv() appends its chunk to the buffer.
When the client closes the connection, recv() returns empty bytes, b"". We stop, because nothing more will ever arrive.

Run the server and call it with curl. The output looks the same as before, because curl sends its whole request at once. The difference only shows up when a request arrives in pieces, and you will try that at the end of this page.

Decode and Parse The Request

Now we have the full request. Let's look at http's request's anatomy once:

GET /index.html HTTP/1.1       <- request line: method, target, version
Host: 127.0.0.1:8000           <- header
User-Agent: curl/8.22.0         <- header
Accept: */*                    <- header
                               <- blank line: end of the headers

The first line is the request line: three values separated by single spaces.
Every other line up to the blank one is a header, written as Name: value.
We call the middle value of the request line the target, not the path, because it can also carry a query string such as /search?q=python.

To split the request into parts we will create a new function parse_request(). Add this function below read_request():

server.py

def parse_request(raw: bytes) -> tuple[str, str, str, dict[str, str]]:
    if b"\r\n\r\n" not in raw:
        raise ValueError("incomplete request")
 
    head = raw.split(b"\r\n\r\n", 1)[0].decode("iso-8859-1")
    lines = head.split("\r\n")
 
    parts = lines[0].split(" ")
    if len(parts) != 3:
        raise ValueError("malformed request line")
    method, target, version = parts
 
    headers = {}
    for line in lines[1:]:
        name, sep, value = line.partition(":")
        if not sep:
            raise ValueError("malformed header")
        headers[name.strip().lower()] = value.strip()
 
    return method, target, version, headers

Let's walk through this function, what exactly is it doing?

Check it is complete. : If there is no blank line, the client hung up halfway, so we raise a ValueError.
Keep only the head. : raw.split(b"\r\n\r\n", 1)[0] takes everything before the blank line. Anything after it would be a request body, which we ignore for now.
Turn bytes into text. : We decode the bytes with iso-8859-1
Split into lines, using "\r\n".
Read the request line. : Splitting on spaces must give exactly three parts. If not, the request is malformed.
Read each header. : partition(":") cuts the line at the first colon. A line with no colon is malformed.
Tidy the headers. : Names are case-insensitive, so we lower-case them, and we strip extra spaces from values.

Run The Server

Now use the parse in our main code:

server.py

import socket

HOST = "127.0.0.1"
PORT = 8000

def read_request(conn: socket.socket) -> bytes:
    data = b""
    while b"\r\n\r\n" not in data:
        chunk = conn.recv(1024)
        if not chunk:  # the client closed the connection
            break
        data += chunk
    return data

def parse_request(raw: bytes) -> tuple[str, str, str, dict[str, str]]:
    if b"\r\n\r\n" not in raw:
        raise ValueError("incomplete request")
 
    head = raw.split(b"\r\n\r\n", 1)[0].decode("iso-8859-1")
    lines = head.split("\r\n")
 
    parts = lines[0].split(" ")
    if len(parts) != 3:
        raise ValueError("malformed request line")
    method, target, version = parts
 
    headers = {}
    for line in lines[1:]:
        name, sep, value = line.partition(":")
        if not sep:
            raise ValueError("malformed header")
        headers[name.strip().lower()] = value.strip()
 
    return method, target, version, headers

def main() -> None:
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as server:
        server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        server.bind((HOST, PORT))
        server.listen()
        print(f"listening on http://{HOST}:{PORT} (Press Ctrl+c to stop)")
        
        try:
            while True:
                conn, addr = server.accept()
                with conn:
                    print(f"connection from {addr}")
                    raw = read_request(conn)
                    try:
                        method, target, version, headers = parse_request(raw)
                    except ValueError as exc:
                        print(f"bad request: {exc}")
                        continue
 
                    print(f"{method} {target} {version}")
                    for name, value in headers.items():
                        print(f"  {name}: {value}")
        except KeyboardInterrupt:
            print("\nbye")

if __name__ == "__main__":
    main()

We are using parse_request() in the main() function, so that it can get methods, target, version and headers then it prints all headers and request line. Notice the header names are lowercase. That's the work of lower() in our parser. Now, start the server and make a curl request from a second terminal. Expected output:

GET / HTTP/1.1
  host: 127.0.0.1:8000
  user-agent: curl/8.22.0
  accept: */*

The stream of bytes has become a method, a target, a version and a tidy dictionary of headers. Now open http://127.0.0.1:8000 in your browser. Here is part of what a browser request looks like after parsing (yours will differ):

GET / HTTP/1.1
  host: 127.0.0.1:8000
  user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:155.0) Gecko/20100101 Firefox/155.0
  accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
  accept-language: en-US,en;q=0.9
  accept-encoding: gzip, deflate, br, zstd
  sec-gpc: 1
  connection: keep-alive
  upgrade-insecure-requests: 1
  sec-fetch-dest: document
  sec-fetch-mode: navigate
  sec-fetch-site: none
  sec-fetch-user: ?1
  priority: u=0, i

Finally, send something that is not HTTP at all. Open second terminal and run:

Terminal

python3 -c "import socket; s = socket.create_connection(('127.0.0.1', 8000)); s.sendall(b'hello\r\n\r\n')"

The server survives, and prints bad request: malformed request line.

What just happened?

When a client connected, accept() returned a socket for that connection, just like before. This time we passed it to read_request(), which kept calling recv() until the blank line (\r\n\r\n) showed up in the buffer. That blank line is how we knew the headers were complete.

Next, parse_request() took those raw bytes, decoded them into text, and split them into lines. The first line became the method, target and version. Every other line was cut at its first colon into a header name and value, and we stored them in a dictionary with lowercase names.

Then main() printed the request line and each header.

When we sent hello\r\n\r\n, the data had a blank line but no valid request line, so parse_request() raised a ValueError. We caught it, printed the problem, and moved on to the next connection instead of crashing.

Try it

Send a request in two pieces, with a pause in the middle:

Terminal

python3 -c "import socket, time; s = socket.create_connection(('127.0.0.1', 8000)); s.sendall(b'GET / HTTP/1.1\r\nHo'); time.sleep(2); s.sendall(b'st: ByteStack\r\n\r\n')"

Your new server waits for the second piece and prints the whole request. Try sending Host: ByteStack as HOST: ByteStack in the above command. Does the parsed output change?

Our server now understands what the client wants, but it still says nothing back, so every client ends up with an empty reply. Next, we write the other half of the conversation: sending an HTTP response.

Read The Complete HeadersCopy Anchor Link

Decode and Parse The RequestCopy Anchor Link

Run The ServerCopy Anchor Link

On this page

Read The Complete Headers

Decode and Parse The Request

Run The Server