Protecting against deeply-nested GraphQL queries

Learn about a denial-of-service queries that attackers can use to overwhelm unprotected GraphQL servers.

How the attack works

Let’s say that an attacker is trying to overwhelm Github’s API to take it down. They don’t have a reason for doing this. They just want to.

They examine Github’s GraphiQL instance and they notice something interesting: Users and repositories recursively refer to each other. This means that for any users, you can get their repositories. For any repositories, you can get its contributors as Users. This means that attackers can create queries that recurse as deep as they want!

The pseudocode for the query would look something like this:

For some repository,
  - Get me all of the contributors for that repository
    - For each of those contributors, get me their repositories
      - For each of those repositories, get me the contributors
        - For each contributor, get me their repositories
          - And so on

Let’s put some numbers to this: Let’s say that repositories have on average 5 contributors, and each contributor has 10 repositories. Here’s how many objects are accessed by the database query per “get the users then get the repositories” round:

• One round: 5 users * 10 repositories = 50 objects
• Two rounds: 50 objects * 5 users * 10 repos = 2500 objects
• Three rounds: 2500 objects * 5 users * 10 repos = 125,000 objects
• Four rounds: 125000 objects * 5 users * 10 repos = 6,250,000 objects

And the attacker doesn’t have to stop here. They can nest it arbitrarily deep. 1000 rounds? Sure, no problem.

This takes down the server by causing the weakest point to either become too slow, or exhausts the memory of one of the components. Perhaps the database becomes overwhelmed from trying to read millions of records. Or maybe the database succeeds in returning enough records without affecting other users, but the web server itself runs out of memory attempting to serialize all of the records into a single JSON response that fits in memory. The attacker just needs to increase the depth until it does. It takes them very little time to make the query deeper.

Here’s roughly what the attack query looks like in GraphQL’s query language.

query { 
  repository(owner:"rails", name:"rails") {
    assignableUsers (first: 100) {
      nodes {
        repositories (first: 100) {
          nodes {
            assignableUsers(first: 100) {
              nodes {
                repositories(first: 100) {
                  nodes {
                    # Iterate until the server is exhausted
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Note that this won’t actually work against GitHub, because they’ve thought about this.

Persisting queries

One mitigation strategy is to use persisted queries.

Normally, GraphQL requests have two payloads: a query and variables.

In JSON format, the query bundle might look like this:

{
  "query": "query {
    user(id: $user_id) {
      name
    }
  }",
  "variables": {
    "user_id": 123 
  }
}

This isn’t the only way to execute GraphQL queries, though. A server could be configured to persist GraphQL queries instead. The client can no longer specify custom GraphQL queries for themselves. Instead, the query itself is defined on the server and referenced by the client.

Here’s an example of what that request might look like:

{
  "query": "SomeHashValueReferringToTheQuery",
  "variables": {
    "user_id": 123 
  }
}

On the server, the handler would look up "SomeHashValueReferringToTheQuery" and substitute it with the appropriate query before passing the query and variables to the GraphQL engine.

This isn’t a silver bullet. Query hashing is challenging to productionize. For example, if a version of your client is released into the wild, then the server needs to understand all hashed queries as long as the app is supported. If your users are still running versions of a mobile app from 2019, then the query hashes from that version of the app need to keep working. This means that you need to store query hashes that are no longer used by any query in your codebase.

Limiting query depth

In this case, your GraphQL server gets an explicit configuration option. This option tells GraphQL to reject any query that contains a greater recursive depth than the option.

For example, the following query has depth 2:

query {
  user(id: $user_id) { # depth 1
    name               # depth 2
  }
}

This is a little easier to manage than the previous method. You can do this by finding the absolute deepest query that you need to execute, and then giving yourself some breathing room on top of it. For example, if the deepest query you execute is 7 deep, then set the maximum depth to 9 or 10. It’s easy to submit a new query that’s deep enough to break it. You could either add a CI check to prevent this. If your reverts are quick enough, you could just plan to revert, fix, and push.

Setting this strict helps protect against other kinds of attacks. For example, an attacker attempting to scrape your website might rely on broad and deep queries to pull as much data as possible. If you only execute really shallow queries, and you set the depth to something really restrictive, then you’ve removed one tool in their toolbelt.

Disabling introspection

A clever engineer might notice that the problem above was caused by introspection. So maybe it’s possible to just disable introspection. In fact, you should definitely turn off introspection in production, unless you manage an open API.

But this is insufficient to stop a motivated attacker. First, they can look through the network traffic that your client sends (or look at strings that are defined within your app) to discover the names of nodes in your schema. They can feed this information into fuzzers that attempt to guess what the schema looks like based on this information. They won’t discover all of the mutation and query definitions that are defined in your server, but for this attack, they only need to find a single recursive one.

Again, you should definitely disable introspection and not serve GraphiQL in production unless you must. It won’t stop a motivated attacker, but by limiting the information they have, you can help multiple mitigation methods work together.

Protect yourself with a general-purpose GraphQL proxy

Not every GraphQL framework offers features to limit query depth. If you are in one of those situations, you can use a proxy server to reject those requests on your behalf. The proxy itself will parse the queries and reject queries that are too deep.

I am an independent software developer, and I am working on a proxy service that will protect against these kinds of attacks. If you’d like to use a proxy service that allows GraphQL servers written in any language to be productionized, please read more at this link and leave your email.