An Introduction to Parser Differentials
Since the arrival of the internet, a growing number of devices have become connected online. This expansion of the attack surface has led to a rapid increase in vulnerabilities, increasing the likelihood of exploitation. As a result, data breaches have become more common. Compared to common vulnerabilities that have caused many data breaches, parser differentials are less known because they are a new niche in vulnerability research. This post will introduce parser differentials in the context of web applications.
What Are Parser Differentials?
A parser differential is a vulnerability that occurs when two parsers interpret the same input differently. A parser is a system component that processes input received from the user of an application. Since web applications are widespread, many parser differential vulnerabilities appear here, and the URL is a common location for them.
The figure below shows how the interpretation of a URL parser differential payload differs between parsers. http://127.0.0.1\@domain.tld is a URL parser differential payload that contains the 127.0.0.1 and domain.tld hostnames. Urlparse parses domain.tld while requests accesses 127.0.0.1.
This can result in a filter bypass if validation is only performed on domain.tld. For instance, in the Python code below, the parsed URL is compared against a blacklist while the same check is not performed on the URL which is later requested.
url = request.args.get("url")
parsed = urlparse(url)
if parsed.hostname == "127.0.0.1":
return "Access to this host is blocked", 403
response = requests.get(url)What Causes Parsers to Interpret the Same Input Differently?
The differences in interpretations of parsers such as urlparse and requests arises as they follow different standards such as the RFCs and WHATWG. This results in each parser parsing a different hostname in the URL. Parser differentials can also occur on the client-side where a browser interprets the URL differently compared to a server-side parser. Despite the validation of the URL by the server-side parser, the browser can end up loading a URL that is unvalidated.
Where Are Parser Differentials Found?
Parser differentials are found in various system components such as parser libraries, reverse proxies and application servers. Parser libraries are third-party software that are used by a web application on the server-side. Parser differentials can arise between more than one parser library or between a parser library and a browser.
Another server-side parser differential can occur between reverse proxies and application servers. Similar to parser libraries, these system components can interpret the same URL differently, although in certain system architectures, the parser differential can appear in a different part of the request.
What Are the Types of Parser Differential Vulnerabilities?
Parser differentials arise on both the client and server-side and include common web application vulnerabilities such as cross-site scripting, open redirect and server-side request forgery. While a parser differential is similar to a common web application vulnerability, the difference lies in the root cause. This is due to the differences in how the parsers interpret user input, which was previously discussed.
Why Are Parser Differential Vulnerabilities Hard to Address?
Due to being a niche area, there is a lack of awareness of parser differentials compared to other web application vulnerabilities. This is evident when examining automated tool rules for finding vulnerabilities, as the focus is on common web application vulnerabilities. For example, when using static analysis to search for URL parser differential vulnerabilities in the source code, this would be missed. This is because the different parser interpretations occur at web application runtime.
How Can Parser Differential Vulnerabilities Be Prevented?
- Perform an inventory of parsers: all parsers in use by a system should be inventoried. This will provide visibility into which parsers are used and how they differ
- Update parser libraries: while inventorying parsers, it is important to apply security updates. These updates can be obtained by following CVE feeds
- Minimise the number of parsers that are used: this will decrease the attack surface by reducing the risk of the same input being interpreted differently
- Normalise user input before validation: before validating user input, it should be normalised to ensure it is in a standard format. This will mitigate parser differential vulnerabilities