Local IP in login log

I want to prevent brute-force login attempts using fail2ban.

Thus my config is set to log failed login attempts:

  default: critical
    homeassistant.components.http.ban: warning

In nginx I am passing the original IP to home-assisstant:

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

The log still wrongfully claims my nginx Ip responsible for the requests:
WARNING (MainThread) [homeassistant.components.http.ban] Login attempt or request with invalid authentication from

How do I fix this?

This seems to be an issue with Docker at the moment. See: