Sorry to hear there are so many challenges with the new hosting service. "No good deed goes unpunished" I guess, with all the improvements you were trying to deliver.
I do not know nginx configuration or behavior, but it certainly sounds like there is a proxy, and a proxy can many times be configured to be transparent to the server. i.e. Even if there is a proxy between your clients and your application server, it can still be configured to pass along the actual client IP address in a way that is "invisible" to any application expecting the client IP address to be the actual source IP address.
Yes, the captcha situation is probably because every failed login attempt looks like "a login attempt coming from the same IP address as every other failed login attempt." Just an additional symptom of the same underlying root cause.
If whatever proxy is in play currently provides the
X-Forwarded-For
HTTP header field, its possible your captcha behavior could be mitigated by setting
"Limit login attempts by X_FORWARDED_FOR header:" to
"Yes" under
ACP, General, Server Configuration, Security Settings. At least this seems to be the intention of this setting. If it doesn't have an effect, realize that we don't actually know that the
X-Forwarded-For
HTTP header field even exists in your configuration; we are just assuming or hoping it might.
There is also related configuration in that same list,
"Validate X_FORWARDED_FOR header:", which intends to match new clients to existing sessions using the
X-Forwarded-For
HTTP header field instead of the IP address the new client request arrived from. Which again, "solves another one of the problems", but doesn't actually "switch the phpBB client IP address value to the one in the
X-Forwarded-For
HTTP header field."
To literally say "use the
X-Forwarded-For
HTTP header field value for everything", it appears to require an extension such as
[BETA] Trust X-Forwarded-For. Keeping in mind, this still assumes the IP address was being passed in
X-Forwarded-For
, which is not something we actually know yet in your specific configuration.
If it were my server, I would continue trying to get the hosting service to update any proxy configuration, such that the proxy is transparent to the running applications; phpBB or otherwise. But while waiting for them to do that, I would test the two existing phpBB configuration settings,
"Limit login attempts by X_FORWARDED_FOR header:" and
"Validate X_FORWARDED_FOR header:".
If those phpBB settings address the primary issues under the current proxy configuration, and the host isn't changing the proxy configuration any time soon, then maybe you want to look into using the
[BETA] Trust X-Forwarded-For. So that you could also see "the real IP address" in other situations such as logged messages, moderation IP address info, etc.