HTTP and HTTPS
From charlesreid1
Let's talk about the web.
Specifically, let's talk about where the traditional "software engineer" web programmer's job ends, and where the (much more interesting) networking portion begins.
Contents
Layers
Computers ultimately speak a single language, composed of two symbols: 0 and 1. The communication of information between computers, however high-level it might be (a YouTube video of cats, say) must ultimately be broken down, chopped up, and send over the wire and across the internet to a server (and vice versa) using voltages - electrical 0s and 1s.
So all the HTML for the pages you're visiting, the Javascript libraries for the fancy graphics, the icons, the background images - all of that was broken into chunks of voltage, 0s and 1s, transmitted hundreds or thousands of miles by wire from switch to switch, to reach your computer. That cascade - from the very high level down to the very low level - is broken down into the canonical seven layers of the network stack (from the Packet Analysis page):
| Layer | Name | Function |
|---|---|---|
| 7 | Application | Topmost layer, provides users a means to access network resources (only level seen by end user) |
| 6 | Presentation | Transforms data received into a format that is readable by the application layer. Handles encryption/description for secure data |
| 5 | Session | Manages communication sessions between computers. Manages connections with other devices. Half-duplex or full duplex. |
| 4 | Transport | Provide reliable data transport services to lower layers. |
| 3 | Network | Routes data between physical networks. Handles addressing, via IP. Handles packet fragmentation and error detection. Router level. Most complex layer. |
| 2 | Data Link | Transports data across a network. Provides addressing scheme to identify physical devices, bridges, switches, MAC addresses. |
| 1 | Physical | The physical medium for the network communication signals. |
(Please Do Not Touch Steve's Pet Alligator)
The chopped-up messages are called packets, and network devices (ethernet ports, wireless cards, network routers) are designed to send and receive packets.
This uses an internet connection and a network of some kind; that network is a TCP/IP network. The web server you connect to uses HTTP, hypertext transmission protocol, is the protocol used to compose the websites we visit in a browser. There are other protocols as well, like SSH and VPN. But how do all these protocols relate? How do you sort out what relates to what?
You can use the Network Stack to construct the right mental model for a network connection, and use it as a roadmap for figuring out how different technologies fit together.
Layers 1 and 2
The hardware is at the bottom layer of the network stack. Layer 1 is the physical connection, and can be thought of as the actual voltages traveling across ethernet wires, or the raw binary data being transmitted by a wireless radio chip. Layer 2 is the MAC address of the hardware. At the MAC address layer, there is no knowledge of network protocols, there is no IP address, and there is no way to communicate with a device - MAC addresses are primarily useful as unique device identifiers, which in turn is useful for higher levels of the networking stack.
Layer 1 = physical layer
Layer 2 = MAC addresses
Layers 3 and 4
Layer 3 deals with how information travels across the network. For the internet, this is IP (internet protocol). The IP protocol uses packet-switched routing to chunk information up and send it out across a network. Thus, Layer 3 is the first layer at which computers "know" about packets and packet-switching networks. IP defines how a packet will travel a long distance - by bouncing from server to server in multiple hops - and defines a "best effort" system whereby packets do their best to get to the destination.
The IP protocol at layer 3 is also where IP address definitions come from - both IPv4 and IPv6. These addresses are used for specifying the source and destination of packets.
Layer 4 is the transport layer. For the internet, this is TCP (transmission control protocol). TCP is complementary to IP, and defines many things that IP doesn't, so they usually go together on a network and are called TCP/IP. TCP defines duplex connections between computers. TCP introduces the concept of a computer port. Computers have thousands of available ports, and these ports can be used with communication streams to send data back and forth between machines.
More concretely, connecting to a particular port on a machine might return the website hosted on that machine. Connecting to a different port might connect you to a database hosted on that machine. Yet another port might connect to other ports, and modify them or perform some logic using the connection.
Layer 3 = IP addresses
Layer 4 = Ports
Layers 5, 6, and 7
IP is a massive improvement over bare, un-networked devices. But there are lots of things it doesn't define. For example, defining protocols to allow packet transmission to happen for multi-channel duplex communications.
Many of those things are defined by TCP, which is a massive improvement over just IP alone. But there are still things that TCP does not define. For example, TCP/IP together define ways of packets getting to and from routes. But they don't define sessions - when and how servers identify and remember particular clients.
Layer 5 is the layer dealing with sessions and states of transmission. You can think of this as the Nmap layer - most Nmap settings and flags deal with some part of Layer 5. Here is a diagram showing the different states a TCP packet can be in, and the transitions it can undergo. Think of this as a conversational flowchart for computers talking to each other across a network:
The next to last layer, Layer 6, is one of the most important, and that is the encryption layer. It's critical that encryption be a high enough layer on the stack that random devices and system software won't have access to unencrypted packets. Packet data, up to Layer 5, reaches the browser at Level 6. The browser will be managing the user's browser session data, and will have the necessary cookies, certificates, private keys, and public keys to decrypt encrypted traffic. Again, this model ensures that nobody but the browser will see unencrypted data.
The final layer, Layer 7, is you and your browser - the final rendered content. This is where the fun of Webapps with Javascript and friends begins.
Layer 5 = Nmap
Layer 6 = Encryption
Layer 7 = Your Browser & You
HTTP and HTTPS Traffic
How does HTTP and HTTPS traffic fit into this seven-layer cake?
HTTP
HTTP is traffic designed for rendering web pages in web browsers. When you want to see a website, you'll type the name into the address bar, and the website will open. But there's a couple of steps happening under the hood.
The browser has to translate your request (typing "charlesreid1.com" in the address bar) from something at Layer 7 down through each layer. First, the website address is translated into an IP address (an address for the server hosting that website). Your computer does a DNS lookup to turn a website address like "charlesreid1.com" into an IP address for the server hosting that website (this is the Layer 3 IP address of the remote server). Then your browser sends a request to that server, on port 80 (using Layer 4 TCP or TCP/IP introduces the concept of ports), and asks to make a connection (using special packet headers like SYN and ACK, part of Layer 5).
Once the connection is established, the contents of the website, which are to be rendered by the browser, are broken up into packets, sent over the network, and arrive at your computer, where they are reassembled as they pass up the various networking Layers.
When the packets reach the final Layer 7, the HTTP layer, at the browser, they are then evaluated by the browser and render the pages that you see and interact with.
Visualizing the Process
We can visualize this HTTP request using Curl and Wireshark.
Curl: Download a Page
First, we can use the Unix curl utility, which is a command-line utility for constructing requests of many protocol types: DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET and TFTP. Let's construct an HTTP request, and print out extra information:
$ curl -v http://charlesreid1.com &> file
Let's take a look at the output and see how Curl constructs the HTTP request.
* Rebuilt URL to: http://charlesreid1.com/
* Hostname was NOT found in DNS cache
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying AA.BB.CC.DD...
* Connected to charlesreid1.com (AA.BB.CC.DD) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.37.1
> Host: charlesreid1.com
> Accept: */*
>
This is where Curl's actual request is printed. This is analogous to what the browser is doing when the user types "charlesreid1.com" in the address bar and hits enter - doing a reverse DNS lookup to find that charlesreid1.com is at an IP addr of AA.BB.CC.DD, and then connecting to that remote machine on port 80.
Once the connection is established, Curl sends a GET request to the server, which is a type of HTTP request that says, "give me a resource."
We can then see the HTTP response of the server, which is a 200 OK (meaning, yes, I have that page, here it is...). There are some other metadata fields that help ensure the content is displayed properly and things are logged and checked and all that.
< HTTP/1.1 200 OK * Server nginx/1.6.2 (Ubuntu) is not blacklisted < Server: nginx/1.6.2 (Ubuntu) < Date: Tue, 01 Sep 2015 17:35:24 GMT < Content-Type: text/html < Content-Length: 12228 < Last-Modified: Wed, 05 Aug 2015 00:58:51 GMT < Connection: keep-alive < ETag: "55c15fcb-2fc4" < Accept-Ranges: bytes [...contents of page snipped...]
Wireshark: Examine Curl Request
If we run Wireshark while we run that Curl request, we see a couple of interesting packets. First is the exchange of packets that establishes the connection between the client and the server. Then we can see