Change file.php to a fixed url for attachements

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Scam Warning
myworkpool
Registered User
Posts: 1
Joined: Mon Oct 30, 2017 5:07 pm

Change file.php to a fixed url for attachements

Post by myworkpool » Tue Oct 31, 2017 1:09 pm

Hello,

I'm currently running a phpbb board with lot of attachments from users and now having a trouble with high CPU usage and 503 Error.

1. Due to every single image file, for avatar and attachment, must be queried from "file.php" which really lock down the performance of the server and php itself. Can the image link be replaced with their actual URL like

"www.abc.com/contents/image_attachment_332.jpeg"

instead of

"www.abc.com/download/file.php?id=332"

2. and in the asset file is it correct that the attachments are stored in "files" folder? Can we use a way to divide them into a date related folder like "2017/10/image_attachment_332.jpeg"

Thanks

thescott
Registered User
Posts: 4
Joined: Mon Mar 12, 2018 4:31 pm

Re: Change file.php to a fixed url for attachements

Post by thescott » Mon Mar 12, 2018 4:33 pm

Why is it even done the other way by default? It is so incredibly inefficient. I don't want the entire PHP engine called every time someone views an image?

The "download/file.php" script is the biggest resource hog in my logs by far. It's wasting so much of my CPU.

Paul
Infrastructure Team Leader
Infrastructure Team Leader
Posts: 25180
Joined: Sat Dec 04, 2004 3:44 pm
Location: The netherlands.
Name: Paul Sohier
Contact:

Re: Change file.php to a fixed url for attachements

Post by Paul » Tue Mar 13, 2018 2:11 pm

thescott wrote:
Mon Mar 12, 2018 4:33 pm
Why is it even done the other way by default? It is so incredibly inefficient. I don't want the entire PHP engine called every time someone views an image?

The "download/file.php" script is the biggest resource hog in my logs by far. It's wasting so much of my CPU.
It is done for security reasons. file.php mostly serves user content, and as such it does several security checks to prevent certain kinds of attacks, especially with older browsers. It also prevents things like code execution (On badly configured servers) as the uploaded files are not directly executed, but the script reads them from the filesystem before sending. File.php also does several permission checks to see if the requested attachement is actually visible for the user who requested it.
Knock knock
Race condition
Who's there?

My BlogMy Photosmy phpBB Extensionscustom phpBB work & Development

User avatar
AmigoJack
Registered User
Posts: 5569
Joined: Tue Jun 15, 2010 11:33 am
Location: グリーン ヒル ゾーン
Contact:

Re: Change file.php to a fixed url for attachements

Post by AmigoJack » Tue Mar 13, 2018 5:09 pm

And is able to count how often it is requested.

If all that is not of your interest then: yes, it's just a resource hog.
The worst thing about censorship is ███████████
Affin wrote:
Tue Nov 20, 2018 9:51 am
The problem is probably not my English but you do not want to understand correctly.
...
We will not come anybody anyway, nevertheless, it's best to shit this.

thescott
Registered User
Posts: 4
Joined: Mon Mar 12, 2018 4:31 pm

Re: Change file.php to a fixed url for attachements

Post by thescott » Tue Mar 13, 2018 5:49 pm

Thank you for the explanation.

I totally get it as being a good idea for attachments generally, but it is a lot of resources for displaying avatar images. On most boards, I imagine the amount of avatar views is hundreds or thousands times more than the number of attachment views.

Granted, if a board admin wants some or all avatars to not be public, then they would indeed have to be dynamically delivered.

Unfortunately, for those admins who would rather serve avatars as static images (much like smilies are served), having file.php called for every single avatar image view is a huge resource hog. Maybe someone will make an extension to help with this.

User avatar
canonknipser
Registered User
Posts: 1955
Joined: Thu Sep 08, 2011 4:16 am
Location: Germany
Name: Frank Jakobs
Contact:

Re: Change file.php to a fixed url for attachements

Post by canonknipser » Tue Mar 13, 2018 6:25 pm

If the traffic loading avatars via file.php is too much for your board, change from user avatars to gallery avatars uploaded by you as a board owner or gravatars, loaded from a remote service, both builtin in phpBB and served as a direct link to the ressource.
Greetings, Frank
phpbb.de support team member
English is not my native language - no support via PM or mail
New arrival - Extensions and scripts for phpBB

Paul
Infrastructure Team Leader
Infrastructure Team Leader
Posts: 25180
Joined: Sat Dec 04, 2004 3:44 pm
Location: The netherlands.
Name: Paul Sohier
Contact:

Re: Change file.php to a fixed url for attachements

Post by Paul » Tue Mar 13, 2018 6:54 pm

The security precautions also apply to uploaded avatars 😊
Knock knock
Race condition
Who's there?

My BlogMy Photosmy phpBB Extensionscustom phpBB work & Development

thescott
Registered User
Posts: 4
Joined: Mon Mar 12, 2018 4:31 pm

Re: Change file.php to a fixed url for attachements

Post by thescott » Wed Mar 14, 2018 3:55 pm

Paul wrote:
Tue Mar 13, 2018 6:54 pm
The security precautions also apply to uploaded avatars 😊
Interesting. Why don't the security precautions apply to remotely linked avatars? I didn't realize non-private jpg files could be so dangerous to serve as static jpg files. I also feel like there must be a way to validate the jpg file once upon uploading and saving to the server, rather than loading the PHP engine and doing the checks every single time the image is served. The same goes for various other image formats like png.

User avatar
AmigoJack
Registered User
Posts: 5569
Joined: Tue Jun 15, 2010 11:33 am
Location: グリーン ヒル ゾーン
Contact:

Re: Change file.php to a fixed url for attachements

Post by AmigoJack » Wed Mar 14, 2018 5:10 pm

thescott wrote:
Wed Mar 14, 2018 3:55 pm
Why don't the security precautions apply to remotely linked avatars?
Because those are linked and thus can't be (malicously) parsed by PHP or whatever.

thescott wrote:
Wed Mar 14, 2018 3:55 pm
I didn't realize non-private jpg files could be so dangerous to serve as static jpg files.
That's a wrong conclusion.

thescott wrote:
Wed Mar 14, 2018 3:55 pm
doing the checks every single time the image is served
The content isn't checked every time. The access permission is, and after that the avatar filedata is passed thru - the avatar file itself is not accessible from the outside.
The worst thing about censorship is ███████████
Affin wrote:
Tue Nov 20, 2018 9:51 am
The problem is probably not my English but you do not want to understand correctly.
...
We will not come anybody anyway, nevertheless, it's best to shit this.

thescott
Registered User
Posts: 4
Joined: Mon Mar 12, 2018 4:31 pm

Re: Change file.php to a fixed url for attachements

Post by thescott » Wed Mar 14, 2018 6:04 pm

AmigoJack wrote:
Wed Mar 14, 2018 5:10 pm
The content isn't checked every time. The access permission is, and after that the avatar filedata is passed thru - the avatar file itself is not accessible from the outside.
Doing it that way is essentially what's needed for a board admin who wants some or all avatars to be private or restricted.

What I'm saying is, for admins who are fine having all avatars publicly viewable, loading the PHP engine (and presumably often doing database queries) to check access permissions each time an avatar is displayed is a needless resource hog. This is especially considering a normal pageview so to speak could trigger 10 or 20 avatars to be viewed.

If one day there was extension that provided a workaround for that to reduce CPU load for admins who have avatars all public, I think it would be a great one.

User avatar
Mick
Support Team Member
Support Team Member
Posts: 21053
Joined: Fri Aug 29, 2008 9:49 am
Location: Watching cricket - definitely

Re: Change file.php to a fixed url for attachements

Post by Mick » Thu Mar 15, 2018 8:40 am

thescott wrote:
Wed Mar 14, 2018 6:04 pm
If one day there was extension that provided a workaround for that to reduce CPU load
Nobody will build an extension if they don’t know about it. You should make an Extension Request.
"The more connected we get the more alone we become" - Kyle Broflovski

reardenlife
Registered User
Posts: 37
Joined: Tue Mar 26, 2019 4:50 am

Re: Change file.php to a fixed url for attachements

Post by reardenlife » Sun Apr 07, 2019 1:27 pm

Having the same problem here: viewtopic.php?f=556&t=2507251#p15232456

Namely, phpbb is using file.php to display attached images (or their thumbs?). So every time the user is loading a thread, their browser is making dozens requests simultaneously and my server has to produce a separate php-fpm process to handle the request (which is about 4 times higher on RAM than apache process). And in addition to it, the whole purpose of apache event module goes down the drain, because the static content which serves via file.php doesn't recognized by apache as static. See https://httpd.apache.org/docs/2.4/mod/event.html
To illustrate this point we can think about the following two situations: serving a static asset (like a CSS file) versus serving content retrieved from FCGI/CGI or a proxied server. The former is predictable, namely the event MPM has full visibility on the end of the content and it can use events: the worker thread serving the response content can flush the first bytes until EWOULDBLOCK or EAGAIN is returned, delegating the rest to the listener. This one in turn waits for an event on the socket, and delegates the work to flush the rest of the content to the first idle worker thread. Meanwhile in the latter example (FCGI/CGI/proxied content) the MPM can't predict the end of the response and a worker thread has to finish its work before returning the control to the listener. The only alternative is to buffer the response in memory, but it wouldn't be the safest option for the sake of the server's stability and memory footprint.
Holy cow! And you guys doing it for security reasons? Now I will edit the code and will be unable to apply your updates because of it. And the whole point of security patches goes down the drain... :D

User avatar
stevemaury
Support Team Member
Support Team Member
Posts: 50405
Joined: Thu Nov 02, 2006 12:21 am
Location: The U.P.
Name: Steve
Contact:

Re: Change file.php to a fixed url for attachements

Post by stevemaury » Sun Apr 07, 2019 3:13 pm

Well, avatars are cached, client-side.

Not support. Moving to Discussion.
For REALLY good and VERY inexpensive hosting CLICK HERE

I can stop all your spam. I can upgrade or update your Board. PM or email me. (Paid support)

reardenlife
Registered User
Posts: 37
Joined: Tue Mar 26, 2019 4:50 am

Re: Change file.php to a fixed url for attachements

Post by reardenlife » Sun Apr 07, 2019 4:21 pm

includes/functions_content.php

Code: Select all

$inline_link = $filename;
...
$thumbnail_link = $filename;
And the server can breathe again.

Now where I can do the same for avatars?

User avatar
Lumpy Burgertushie
Registered User
Posts: 66230
Joined: Mon May 02, 2005 3:11 am
Contact:

Re: Change file.php to a fixed url for attachements

Post by Lumpy Burgertushie » Sun Apr 07, 2019 4:30 pm

I am no coder but will that actually change what happens when an attachment is called? won't it still have to go through all the steps mentioned above anyway?

also, I wonder why this is such an issue for you. there are many people with very many attachments that do not seem to have this problem of cpu usage.
I just worked on a board that had over 16,000 attachments in the files folder and they did not have these kinds of problems.

the only problem I have seen discussed her along these lines is what you mentioned about splitting up the files into separate folders.


robert
I am available for custom work on a donation basis. Please send me a PM with your needs.

Premium phpBB 3.2 Styles by PlanetStyles.net

OK, so what's the speed of dark?

Locked

Return to “phpBB Discussion”