Does the Extension Manager "re-check all versions" still work?

Get help with installation and running phpBB 3.2.x here. Please do not post bug reports, feature requests, or extension related questions here.
User avatar
hurghanico
Registered User
Posts: 113
Joined: Mon May 07, 2018 10:59 am

Re: Does the Extension Manager "re-check all versions" still work?

Post by hurghanico » Wed Dec 04, 2019 9:21 pm

Update:

I received a reply from the AlterVista forum support guys.

Here it is (translated to english):

"After the necessary checks, AlterVista sends correctly the requests to their server.

The only detail that can affect is that the user agent set by the application is overwritten by an AlterVista one.
If they perform checks on the user agent, this is most likely the problem."

User avatar
EA117
Registered User
Posts: 1181
Joined: Wed Aug 15, 2018 3:23 am
Contact:

Re: Does the Extension Manager "re-check all versions" still work?

Post by EA117 » Thu Dec 05, 2019 2:51 am

hurghanico wrote:
Wed Dec 04, 2019 9:21 pm
"After the necessary checks, AlterVista sends correctly the requests to their server.

The only detail that can affect is that the user agent set by the application is overwritten by an AlterVista one.
If they perform checks on the user agent, this is most likely the problem."
Great, we'll have to see if this ends up factoring into anything that Paul comes up with. That phpbb.com might have been detecting this extension version check "as though it was some kind of bot" was one of the possibilities, and the user agent name is one of the primary way such things are identified. But it's not clear why the agent name would have been set to something that phpbb.com would want to reject.

We'll have to see if phpbb.com confirms the request is being rejected because of its agent name, versus being rejected due to some other criteria.

User avatar
david63
Registered User
Posts: 16845
Joined: Thu Dec 19, 2002 8:08 am
Location: Lancashire, UK
Name: David Wood
Contact:

Re: Does the Extension Manager "re-check all versions" still work?

Post by david63 » Thu Dec 05, 2019 7:02 am

My view after that response is that it is a hosting problem. No other host, that I have heard of, does this and it would appear to be non standard.

As I said before these are the types of problems that you can encounter with free hosts and in the long term I would suggest that you find a new host because experience has shown that if nothing else free hosts have a habit of disappearing overnight.
David
Remember: You only know what you know and - you don't know what you don't know!
My CDB Contributions | How to install an extension
I will not be accepting translations for any of my extensions in Github - please post any translations in the appropriate topic.
No support requests via PM or email as they will be ignored

User avatar
3Di
Former Team Member
Posts: 14487
Joined: Mon Apr 04, 2005 11:09 pm
Location: Milan (IT) Frankfurt (DE)
Name: Marco
Contact:

Re: Does the Extension Manager "re-check all versions" still work?

Post by 3Di » Thu Dec 05, 2019 7:17 am

david63 wrote:
Thu Dec 05, 2019 7:02 am
My view after that response is that it is a hosting problem.
I fully agree. I wanted to say it many posts ago.
david63 wrote:
Thu Dec 05, 2019 7:02 am
No other host, that I have heard of, does this and it would appear to be non standard.

As I said before these are the types of problems that you can encounter with free hosts and in the long term I would suggest that you find a new host because experience has shown that if nothing else free hosts have a habit of disappearing overnight.
Yeah, that's it.

I would however like to add that the host in question has been active for about 20 years if not more, I remember using it for my first experiments as free. However, it is very restrictive and almost dictatorial, I do not suggest it at all. I also suggest it less than many others on the market.
Please PM me only to request paid works. Thx.
Want to compensate me for my interest? Donate
My development's activity º PhpStorm's proud user
Extensions, Scripts, MOD porting, Update/Upgrades
✒️ Black Friday 2019 @ The Studio ▪️◾️

User avatar
hurghanico
Registered User
Posts: 113
Joined: Mon May 07, 2018 10:59 am

Re: Does the Extension Manager "re-check all versions" still work?

Post by hurghanico » Thu Dec 05, 2019 11:07 am

If the checks by phpbb.com will confirm that the problem is caused by the user agent replaced by AlterVista, I wonder what the solution might be (if any) in that case, at least on my part..

..and however I still wonder why the version check of the extensions has always worked well with all the phpBB3 older versions previously installed; as has been suggested it could be a coincidence between my installation of the latest version of the board and a behavior change of the altervista host, but it seems unlikely to me, but you never know...

I will ask in the AlterVista forum if the user agent replacement trick has been implemented recently or if it has been always like this, and I will report back.

I guess that unfortunately it is not possible to revert the board back to the previous version 3.2.7, otherwise I would have done also this test if it was 100% safe to do.

User avatar
EA117
Registered User
Posts: 1181
Joined: Wed Aug 15, 2018 3:23 am
Contact:

Re: Does the Extension Manager "re-check all versions" still work?

Post by EA117 » Thu Dec 05, 2019 2:38 pm

3Di wrote:
Thu Dec 05, 2019 7:17 am
david63 wrote:
Thu Dec 05, 2019 7:02 am
My view after that response is that it is a hosting problem.
I fully agree.
And I don't think you should expect anyone to be surprised by these assertions. We're probably expecting a couple more responses that volunteer a willingness to blame anything else for a problem, even before knowing what the problem actually is yet. Yes, if we wanted to just blame something and make a root cause declaration without the evidence, we could have done that within the first couple messages and not bothered asking about the details.

We've just learned that AlterVista says they change the user agent on outgoing requests, like the outgoing traffic passes through some kind of reverse proxy or similar. Does phpbb.com intend "these customers should not be allowed to check phpBB extension versions?" Or that XAMPP customers sitting behind an ISP who does anything similar should not be allowed? Or that phpbb.com should allow such customers to check their phpBB version, but should not allow them to check the versions of extensions?

The answer is ostensibly "No" to these questions. But before we can understand "why" its happening, we have to know "what" is happening. An assertion of "I don't think AlterVista is a good hosting choice" is not the same thing as "what phpbb.com was doing is intentional", nor "what phpbb.com is doing is necessary", nor that "phpbb.com is unwilling to fix this problem." But again, we have to actually know what the issue is first.

It is possible that phpbb.com is reacting to a specific agent name. Or it is possible that phpbb.com is reacting to the IP network the request is coming from. Or it is possible that phpbb.com is reacting to something none of us here were smart enough to predict. Since neither phpBB nor Tatiana is blatantly written to throw HTTP_GONE / 410 in response to a bots table match or anything else, we're still waiting to learn why 410 was being returned in this case -- specifically and only for the extension version checks -- and what criteria was used in deciding to return it.

The /phpbb/file_downloader.php actually doesn't set any agent name at all on its request, and writes its own very simple HTTP 1.0 GET request. So an agent name has been "added", if anything. But we do at least know that both phpbb.com and Tatiana will accept an agent name being set on the extension version check requests, even though phpBB itself would not have set one. Since when each of us clicks on http://www.phpbb.com/customise/db/extension/mediaembed/version_check the request is accepted, despite the wide and varied range of agent identifications that would be sent when doing this.

David, do you happen to have seen from the logs of your server (shortly after you resolved the .htaccess issue) that any "unique" agent names were logged as successful GET requests for the Privacy Policy version check path? Since we haven't heard back from phpbb.com yet, I'm just wondering if we can know what this agent name is that appears on the AlterVista traffic actually is, in order to possibly predict what it might be matching or identified as. Note we're not wanting to expose publicly here what the requesting IP address is, since OP's server is currently behind Cloudflare.

Or hurghanico, if you're still talking to AlterVista, maybe that's something you could have them answer too: What is the agent name they're asserting into your phpBB server's request. Though the answer from phpbb.com's or David's logs would be more definitive as to the agent name which ultimately arrived. Knowing that agent name would let us perform an independent test from outside AlterVista, to confirm or rule out that the agent name causes the issue.

hurghanico wrote:
Thu Dec 05, 2019 11:07 am
If the checks by phpbb.com will confirm that the problem is caused by the user agent replaced by AlterVista, I wonder what the solution might be (if any) in that case, at least on my part..
It might have been something simply "lost in translation", but if they were looking at the actual outgoing traffic, I'm surprised they would say "replaced" the agent name. Instead of "added" an agent name, since they would have seen that the phpBB's original request simply didn't have an agent name specified at all.

It probably will not make any difference, and they may indeed "replace" the agent name no matter if there was an existing one or not. But you could apply the attached /phpbb/file_downloader.php which at least sets a "User-Agent: phpBB file_downloader" HTTP header to the request. On the off-chance they only supplied an agent name because there wasn't one.
Attachments
file_downloader.php
(2.77 KiB) Downloaded 3 times

User avatar
hurghanico
Registered User
Posts: 113
Joined: Mon May 07, 2018 10:59 am

Re: Does the Extension Manager "re-check all versions" still work?

Post by hurghanico » Thu Dec 05, 2019 3:25 pm

EA117 wrote:
Thu Dec 05, 2019 2:38 pm
It might have been something simply "lost in translation", but if they were looking at the actual outgoing traffic, I'm surprised they would say "replaced" the agent name. Instead of "added" an agent name, since they would have seen that the phpBB's original request simply didn't have an agent name specified at all.
Literally they didn't say "replaced" but "overwritten" in regard to the user agent.
EA117 wrote:
Thu Dec 05, 2019 2:38 pm
Or hurghanico, if you're still talking to AlterVista, maybe that's something you could have them answer too: What is the agent name they're asserting into your phpBB server's request.
I will ask them which is the user agent we are speaking about
EA117 wrote:
Thu Dec 05, 2019 2:38 pm
It probably will not make any difference, and they may indeed "replace" the agent name no matter if there was an existing one or not. But you could apply the attached /phpbb/file_downloader.php which at least sets a "User-Agent: phpBB file_downloader" HTTP header to the request. On the off-chance they only supplied an agent name because there wasn't one.
I will try to apply the attached file_downloader.php. Thanks!

User avatar
hurghanico
Registered User
Posts: 113
Joined: Mon May 07, 2018 10:59 am

Re: Does the Extension Manager "re-check all versions" still work?

Post by hurghanico » Thu Dec 05, 2019 3:39 pm

Report

After having backup the original one I uploaded the kindly supplied file_downloader.php into the phpbb folder.

I purged both the board and the cloudlfare caches and waited a little, then I checked again all the versions: board and extensions..

Nothing changed. Same behavior as before.

User avatar
david63
Registered User
Posts: 16845
Joined: Thu Dec 19, 2002 8:08 am
Location: Lancashire, UK
Name: David Wood
Contact:

Re: Does the Extension Manager "re-check all versions" still work?

Post by david63 » Thu Dec 05, 2019 6:41 pm

The bottom line here is that there is nothing inherently wrong with the process, as has been proven by access to my version control.

There appears to be a problem with accessing phpBB therefore the correct procedure is to file a bug report. There is nothing to be gained by prolonging this topic.

I am out of this now.
David
Remember: You only know what you know and - you don't know what you don't know!
My CDB Contributions | How to install an extension
I will not be accepting translations for any of my extensions in Github - please post any translations in the appropriate topic.
No support requests via PM or email as they will be ignored

User avatar
Mick
Support Team Member
Support Team Member
Posts: 21736
Joined: Fri Aug 29, 2008 9:49 am
Location: Cardiff

Re: Does the Extension Manager "re-check all versions" still work?

Post by Mick » Thu Dec 05, 2019 8:20 pm

I’ll admit I haven’t read all of this but I noticed one thing, using the url you gave originally http://allabouthd500x.altervista.org/ gives me a not found error but it seems you have a redirect that points to https://allabouthd500x.altervista.org/allabouthd500x/ which Is the correct address and always works as expected. Maybe there’s an issue there?
"The more connected we get the more alone we become" - Kyle Broflovski

User avatar
hurghanico
Registered User
Posts: 113
Joined: Mon May 07, 2018 10:59 am

Re: Does the Extension Manager "re-check all versions" still work?

Post by hurghanico » Thu Dec 05, 2019 9:05 pm

david63 wrote:
Thu Dec 05, 2019 6:41 pm
The bottom line here is that there is nothing inherently wrong with the process, as has been proven by access to my version control.

There appears to be a problem with accessing phpBB therefore the correct procedure is to file a bug report.
This is what I will probably do at this point.

User avatar
hurghanico
Registered User
Posts: 113
Joined: Mon May 07, 2018 10:59 am

Re: Does the Extension Manager "re-check all versions" still work?

Post by hurghanico » Thu Dec 05, 2019 9:08 pm

Mick wrote:
Thu Dec 05, 2019 8:20 pm
I’ll admit I haven’t read all of this but I noticed one thing, using the url you gave originally http://allabouthd500x.altervista.org/ gives me a not found error but it seems you have a redirect that points to https://allabouthd500x.altervista.org/allabouthd500x/ which Is the correct address and always works as expected. Maybe there’s an issue there?
Who knows?.. Maybe..
But why the previous phpBB3 versions didn't have the same problem with my board?

User avatar
hurghanico
Registered User
Posts: 113
Joined: Mon May 07, 2018 10:59 am

Re: Does the Extension Manager "re-check all versions" still work?

Post by hurghanico » Thu Dec 05, 2019 10:28 pm

EA117 wrote:
Thu Dec 05, 2019 2:38 pm
Or hurghanico, if you're still talking to AlterVista, maybe that's something you could have them answer too: What is the agent name they're asserting into your phpBB server's request. Though the answer from phpbb.com's or David's logs would be more definitive as to the agent name which ultimately arrived. Knowing that agent name would let us perform an independent test from outside AlterVista, to confirm or rule out that the agent name causes the issue.
Ok, they just told me that the user agent is:

Code: Select all

<your site>.altervista.org/<your IP> (compatible; MSIE 7.0)
They also kindly gave me the following script to verify by myself the user agent, but I have no idea how to use it:

Code: Select all

 <?php
if(isset($_GET['u']))
    echo $_SERVER['HTTP_USER_AGENT'];
else
    echo file_get_contents("http://$_SERVER[HTTP_HOST]$_SERVER[REQUEST_URI]?u=1"); 

User avatar
EA117
Registered User
Posts: 1181
Joined: Wed Aug 15, 2018 3:23 am
Contact:

Re: Does the Extension Manager "re-check all versions" still work?

Post by EA117 » Thu Dec 05, 2019 11:45 pm

hurghanico wrote:
Thu Dec 05, 2019 10:28 pm
Ok, they just told me that the user agent is:

Code: Select all

<your site>.altervista.org/<your IP> (compatible; MSIE 7.0)
It's the "MSIE 7.0". Querying against phpbb.com's Tatiana with the "MSIE" (Microsoft Internet Explorer) version in the User-Agent set to anything less than 9.0 causes the HTTP 410 response.

I can make my own phpBB extension version check fail by changing the "User-Agent: phpBB file_downloader" in the file_downloader.php I attached here to set "User-Agent: phpBB file_downloader (MSIE 7.0)" instead. Advancing to "User-Agent: phpBB file_downloader (MSIE 8.0)" still fails with HTTP 410, but advancing again to "User-Agent: phpBB file_downloader (MSIE 9.0)" successfully retrieves the JSON version check data:

Code: Select all

Hypertext Transfer Protocol
    GET /customise/db/extension/mediaembed/version_check HTTP/1.0\r\n
    HOST: www.phpbb.com\r\n
    User-Agent: phpBB file_downloader (MSIE 7.0)\r\n
    Connection: close\r\n

Hypertext Transfer Protocol
    HTTP/1.1 410 Gone\r\n
    Server: Varnish\r\n
    Content-Type: text/html; charset=utf-8\r\n
    Retry-After: 5\r\n
    Content-Length: 374\r\n
    Accept-Ranges: bytes\r\n
    Date: Thu, 05 Dec 2019 23:34:07 GMT\r\n
    X-Varnish: 1630213066\r\n
    Age: 0\r\n
    Via: 1.1 varnish\r\n
    Connection: close\r\n
    X-Cache: MISS\r\n

Hypertext Transfer Protocol
    GET /customise/db/extension/mediaembed/version_check HTTP/1.0\r\n
    HOST: www.phpbb.com\r\n
    User-Agent: phpBB file_downloader (MSIE 8.0)\r\n
    Connection: close\r\n

Hypertext Transfer Protocol
    HTTP/1.1 410 Gone\r\n
    Server: Varnish\r\n
    Content-Type: text/html; charset=utf-8\r\n
    Retry-After: 5\r\n
    Content-Length: 374\r\n
    Accept-Ranges: bytes\r\n
    Date: Thu, 05 Dec 2019 23:34:26 GMT\r\n
    X-Varnish: 1630213243\r\n
    Age: 0\r\n
    Via: 1.1 varnish\r\n
    Connection: close\r\n
    X-Cache: MISS\r\n

Hypertext Transfer Protocol
    GET /customise/db/extension/mediaembed/version_check HTTP/1.0\r\n
    HOST: www.phpbb.com\r\n
    User-Agent: phpBB file_downloader (MSIE 9.0)\r\n
    Connection: close\r\n

Hypertext Transfer Protocol
    HTTP/1.1 200 OK\r\n
    Server: Apache\r\n
    X-Frame-Options: SAMEORIGIN\r\n
    Set-Cookie: phpbb3_1fh61_u=1; expires=Sat, 04-Jan-2020 23:34:51 GMT; path=/; domain=.phpbb.com; secure; HttpOnly\r\n
    Set-Cookie: phpbb3_1fh61_k=; expires=Sat, 04-Jan-2020 23:34:51 GMT; path=/; domain=.phpbb.com; secure; HttpOnly\r\n
    Set-Cookie: phpbb3_1fh61_sid=888bfc16700fcf29f2c2914500de3ca0; expires=Sat, 04-Jan-2020 23:34:51 GMT; path=/; domain=.phpbb.com; secure; HttpOnly\r\n
    Cache-Control: no-cache\r\n
    Content-Type: application/json\r\n
    X-Balancer: lb1\r\n
    X-Cacheable: YES\r\n
    Content-Length: 217\r\n
        [Content length: 217]
    Accept-Ranges: bytes\r\n
    Date: Thu, 05 Dec 2019 23:34:51 GMT\r\n
    X-Varnish: 1630213441\r\n
    Age: 0\r\n
    Via: 1.1 varnish\r\n
    Connection: close\r\n
    X-Cache: MISS\r\n
    File Data: 217 bytes
JavaScript Object Notation: application/json
    Object
        Member Key: stable
            Object
                Member Key: 3.2
                    Object
                        Member Key: current
                            String value: 1.1.1
                            Key: current
                        Member Key: download
                            String value: https://www.phpbb.com/customise/db/download/168616
                            Key: download
                        Member Key: announcement
                            String value: https://www.phpbb.com/customise/db/extension/mediaembed/
                            Key: announcement
                        Member Key: eol
                            Null value
                            Key: eol
                        Member Key: security
                            False value
                            Key: security
                    Key: 3.2
            Key: stable
So two things I would take from that:
  • If AlterVista said that their adding the User-Agent on the outbound requests wasn't a new behavior, maybe check with them whether the MSIE version included in the User-Agent they added is new or different than it was before. And, of course, ask them whether they have any objection to setting it to MSIE 9.0 or higher instead. 😜
  • If you're adding a bug against the phpbb.com web site in the tracker, you can now tell them specifically that the HTTP 410 is being reported when the HTTP User-Agent of the request includes "MSIE" but with a version lower than "MSIE 9.0". If "MSIE" isn't present, or if no User-Agent is included at all, the request is allow to succeed.


david63 wrote:
Thu Dec 05, 2019 6:41 pm
I am out of this now.
Fair enough. Appreciate again all the help thus far. It was very fortuitous for your extensions to have an independent version check server. That provided a data point key to understanding the scope of the issue more quickly.

User avatar
hurghanico
Registered User
Posts: 113
Joined: Mon May 07, 2018 10:59 am

Re: Does the Extension Manager "re-check all versions" still work?

Post by hurghanico » Fri Dec 06, 2019 1:45 am

EA117 wrote:
Thu Dec 05, 2019 11:45 pm
If AlterVista said that their adding the User-Agent on the outbound requests wasn't a new behavior, maybe check with them whether the MSIE version included in the User-Agent they added is new or different than it was before. And, of course, ask them whether they have any objection to setting it to MSIE 9.0 or higher instead. 😜
Excellent!

thanks to your precious collaboration and expertise the fog is thinning and now we see clearly what causes the error, but we still don't know which side a recent change was made that allowed it to occur, AlterVista or phpbb.com's Tatiana.

I'm still waiting from AlterVista the answer to know if their adding the User-Agent on the outbound requests is a new and recent behavior.
Surely I'll report back here as soon as I get a reply.

However given that phpbb.com's Tatiana refuses queries with "MSIE" version in the User-Agent set to anything less than 9.0, ie basically with MSIE outdated versions, I'm more inclined to believe that the recent behavior change has been made on Tatiana side.

And I am inclined to believe so also because I see the AlterVista low tendency to update themselves and the conseguent extreme slowness with which the new phpBB3 versions are proposed for installation in their control panel, from there it appears to them that I still have v3.1.8 and they have been proposing the update to 3.2.4 for months.

I have not yet added a bug against the phpbb.com web site in the tracker, I am undecided whether to proceed or not.
Surely I would like to know if recent behavior change has been made on Tatiana side.

Post Reply

Return to “[3.2.x] Support Forum”