Getting the “Real IP” from Behind Multiple Proxies in Nginx

Photo by Philipp Potocnik on Unsplash

The org I currently work for has a microservice-based architecture, so of course there needs to be a reverse proxy in front of that to manage the web traffic. Originally we just exposed that proxy directly to the internet [after some security hardening, ofc] and all was well. If we wanted the “Real” user IP address we just referenced $remote_addr in the nginx config.

However, now that we’ve got a mind [and budget] to begin to focus on scalability and fault tolerance at this layer we made the call to stand up more than one of these boxes, and throw them behind an ELB. This seemed like a simple exercise until we looked at the logs and found that all of logged IPs were from the ELB addresses, and our analytics were not going to be happy.

At least for the GeoIP extension this was a simple fix. They were cognizant of proxy headers when they wrote the extension and included the geoip_proxy and geoip_proxy_recursive directives. Just tack on the below and GeoIP will pare off trusted proxies and return results from the canonical source IP:

geoip_proxy 10.0.0.0/8;
geoip_proxy 172.16.0.0/12;

“Wow,” I thought, “If it’s this simple for an extension, then it must be at least equally easy for core config, right?”

“Lol, nah.” I’ve bothered a fair few nginx pros over the last day and there does not seem¹ to be anything for this beyond the nonsense I am about to write below.

The absolute simplest solution is adding the following to the http {} block, likely in your main nginx.conf file:

map $proxy_add_x_forwarded_for $x_real_ip {
  "~^([^,]+).*" $1;
  default $remote_addr;
}

All this does is peel off the first IP address for the X-Forwarded-For: header, which is all well and good if you don’t mind users spoofing other IP addresses via header injection.

Let’s construct an example. My IP is 1.2.3.4, but I want to pretend that I’m 5.6.7.8. I inject my own X-Forwarded-For: header via browser extension and the nginx box gets:

X-Forwarded-For: 5.6.7.8, 1.2.3.4, 10.42.0.1, 172.16.0.1

Ideally we only want to strip off the two trusted proxies and use the first untrsuted IP. For this we’re going to have to get a little creative with the regular expressions:

map $proxy_add_x_forwarded_for $x_real_ip {
  "~(?:^|, )([^,]+), (?:10\.|172\.(?:1[6-9]|2[0-9]|3[01])\.).*" $1;
  default $remote_addr;
}

Breaking that regex out with comments:

(?:                   # Non-capturing group
  ^ |                 # Line beginning, or ..
  ,_                  # Literal comma and space
)
([^,]+)               # Capture group, one or more non-comma chars
,_                    # Literal comma and space
(?:
  10\. |              # 10. or
  172\.(?:            # 172. followed by ...
    1[6-9] | 2[0-9] | 3[01] # A number between 16 and 31, defined the hard way
  )\.                 # Literal period
)
.*                    # The rest of the line, about which we do not care.

It’s a bit extra messy since the 172.x.x.x private network mask isn’t evenly aligned by octet, but also I could have made better network choices in the past. 🤷‍♂️

While this is far better that the first attempt, it’s not perfect. The regex is still reading left-to-right and is looking for the IP preceding the first trusted proxy. If I were an attacker and I knew this kludgey config were in use and I also knew what the trusted networks were I could set the following header to get around it:

X-Forwarded-For: 5.6.7.8, 172.31.0.1

Which would wind up being the following as far as nginx sees:

X-Forwarded-For: 5.6.7.8, 172.31.0.1, 1.2.3.4, 10.42.0.1, 172.16.0.1

Unfortunately there’s no good way to accomplish this via a regular expression, as this is a case that needs a real parser and logic behind it. However, the case in which this can be exploited is so narrow, and the consequences [specifically for my use case] if it were are negligible at worst. YMMV, and if you have improvements please let me know.

The “real” solution to this would be an nginx patch or module that implements the same logic that GeoIP has, but in a more general manner. But I don’t think I’ll hold my breath for that.

Caveat:

• You may get an error saying “you should increase map_hash_bucket_size” which means that that you need to increase that value to accommodate that bigass regex. However, the docs on that are a bit fiddly and talk about “alignment” being important, so if you’ve not otherwise set that value somewhere else I would suggest doubling the value referenced in the message. In my case I doubled it from 64 to 128.

[1] Please prove me wrong on this. I am listening.