Lesson 2 of 2
Requests the server makes
Listen to an AI summary of Requests the server makes
A link preview in the sports shop
On the sports shop website, a customer can write a message to the shop staff. A customer pastes a link into a message, and the shop shows a preview of the link.
A link holds a URL, such as https://news.example/story. The shop's server sends a request to the URL and shows the customer the text of the page.
The Python code below runs on the server and makes the preview. Look at where url comes from and where url goes.
preview.The variable request holds the request from the customer's browser. requests, with an s, is a Python library. The function requests.get sends a request to a URL. No line checks url.
What the server can reach
An internal network is a network that only the computers of one organisation can reach. The server of the sports shop is inside the internal network of the office that runs the shop.
A private address is an IP address that works only inside an internal network, such as 10.0.0.5. An IP address is the number that identifies one computer on a network. A computer outside the office cannot send a request to 10.0.0.5.
The shop keeps a stock page for staff at http://10.0.0.5/stock. The stock page asks for no password, because only computers inside the office can reach the page.
A visitor and the receptionist
A visitor to a sports centre cannot walk into the staff corridor. The visitor asks the receptionist to fetch a folder from the corridor, and the receptionist does.
The visitor is the attacker. The receptionist is the server, and the staff corridor is the internal network. A receptionist would ask why. A server sends a request to whatever address the code passes on.
The attack on the sports shop
An attacker at home pastes the link http://10.0.0.5/stock into a message. The server is inside the office, so the request reaches the stock page. The server then sends the text of the stock page to the attacker.
Attacker
At home, outside the office
- A link in a message
The shop's server
Inside the office, sends a request to the link
- A request from inside
Stock page
10.0.0.5, no password
What is server-side request forgery
Server-side request forgery (CWE-918) is an attacker making the server send a request to an address the attacker chose. The short name is SSRF.
Any feature that fetches a URL from a request can have the weakness. Two more examples are a photo imported from a URL and a web page saved as a PDF file.
The metadata address
A website can run on a server rented from a hosting company, such as Amazon Web Services. The metadata address is a fixed private address, 169.254.169.254, that the hosting company provides. A request from the server to the metadata address returns details about the server.
The details can include a secret, such as a password. The lesson on secrets in code has more on secrets. An attacker who makes the server fetch the metadata address can read the secret.
What is DNS
A check that refuses any URL with 10.0.0. in the text is not enough. A private address can be behind a host name, the name of a computer, such as news.example.
https://news.example/story
- httpsthe scheme, the first part of a URL
- news.examplethe host name
- /storythe path to the page
DNS is the service that turns a host name into an IP address. The owner of a host name chooses the IP address that DNS gives.
An attacker owns the host name photos.evil.example and sets the IP address that DNS gives to 10.0.0.5. The link https://photos.evil.example/stock has no private address in the text. The server looks the host name up, gets 10.0.0.5 and sends the request to the stock page.
The check belongs on the IP address that DNS gives, and not on the host name.
Three steps before the request
The safer code does three steps before the server sends a request.
1.Split the URL
A URL parser finds the scheme and the host name
2.Look the host name up
DNS gives an IP address
3.Check the IP address
Refuse an address that is not public
The Python code below is the safer version of preview. Look at the two lines that start with if.
preview.urlparse is the URL parser in Python, and socket.gethostbyname asks DNS for the IP address. The first if permits only the scheme https. Some functions that fetch a URL also accept the scheme file, which reads a file on the server.
Addresses the server refuses
In the second if, ipaddress.ip_address(ip) turns the text in ip into an IP address Python can check. is_global is True only for a public address on the internet. A check of is_private in its place misses the last row of the table. The if refuses each kind of address in the table:
- A private address
- An address that starts
10.or192.168., or172.16.up to172.31.. - The server's own address
- An address that starts
127., such as127.0.0.1. A request to127.0.0.1goes to the computer that sent the request. - The metadata address
169.254.169.254, and any other address that starts169.254..- A shared address
- An address from
100.64.to100.127., which hosting and phone companies use inside their own networks. One hosting company keeps its metadata address there.
Redirects
A redirect tells the sender to go to a different URL. A public website that passes the check can answer with a redirect to http://10.0.0.5/stock.
The words allow_redirects=False tell requests.get to follow no redirect. Code that must follow a redirect checks each new URL with the same three steps.
What the server sends back
The first version of preview sends back the whole page. The safer version sends back only the title, which the function title_of finds in the page.
An error message such as "connection refused" tells an attacker whether a computer exists at an address. A safer feature sends one short message for every failure.
Examples of other fixes
- A choice in place of an address. The customer picks a name from a list, and the server holds the URL for each name.
- An allowlist of host names, where the feature needs only a few websites.
- A request sent to the IP address that passed the check. The safer code looks the host name up twice, and an attacker can give a different address the second time.
- A check of every address the host name has.
gethostbynamereturns one address of the older kind, named IPv4.
Examples of what to look for
- A new line that sends a request to a URL read from a request, such as
requests.get(url). - A check on the text of a URL, such as
url.startswith("https://news.example"), with no URL parser. - A check of the host name with no check of the IP address that DNS gives.
- A request that follows redirects after the first URL passed the check.
- The whole page that the server fetched, or the error message, sent back to the user.
- A setting where a user types an address for the server to send messages to, named a webhook.
Example words to search for are requests.get and urlopen in Python. Each name sends a request. The lesson on searching the code shows how to search for a word.
Check your understanding
Name two kinds of address the safer version of
previewrefuses.Show the answer
Any two of a private address, the server's own address and the metadata address.
Identify the line in the first version of
previewthat sends a request to a URL from the customer's request.Show the answer
The line
page = requests.get(url). The variableurlholds the link the customer pasted.Give one reason why the check is made on the IP address and not on the host name.
Show the answer
The owner of a host name chooses the IP address that DNS gives. An attacker's host name can lead to a private address.
Practise it
Challenges with a weakness this lesson teaches.
- Go live checklistSwiftcart, IntermediateOpen
- Link previewsChirp, IntermediateOpen