Could anyone help with attachments subfolder MOD ?

Discussion forum for MOD Writers regarding MOD Development.
wile
Registered User
Posts: 59
Joined: Mon Apr 30, 2007 4:21 pm

Could anyone help with attachments subfolder MOD ?

Post by wile »

Hello everyone,

Based on repeating messages from my hosting provider about too many files in 1 folder (root/files/ of course) I am trying to solve this issue that potentially can slow down backing up process or cause any other unexpected situation related to file manipulations via FTPs , etc.

Background of this topic Goal
  • identify good approach how to solve this issue (there are already opened some topics and developed something)
  • develop/put together working/tested solution
Subfolders are a bit sensitive topic as there should be only one mechanism how files are organised - taking into consideration future updates and preventing broken links, missing files, etc.


Below I am attaching my testing results of 3 MODs in development. I am not an author, I just did testing.
Last edited by wile on Tue Dec 24, 2013 12:50 pm, edited 1 time in total.
wile
Registered User
Posts: 59
Joined: Mon Apr 30, 2007 4:21 pm

Re: Could anyone help with attachments subfolder MOD ?

Post by wile »

[RC] Attachments in subfolders
viewtopic.php?f=70&t=2130170&start=60
So, the attachments with ID between 1 and 99 are stored on folder /files/00/00/, the attachments with ID between 100 and 199 are stored on folder /files/00/01/, the attachments with ID between 200 and 299 are stored on folder /files/00/02/... then the attachments with ID between 10000 and 10099 are stored on folder /files/01/00/, the attachments with ID between 10100 and 10199 are stored on folder /files/01/01/, the attachments with ID between 10200 and 10299 are stored on folder /files/01/02/ etc.
Known issues :
- uses attachment IDs to recognise in what folder attachment is supposed to be
- copying of thread potentially causes missing attachments
- after copying there are created new items in attachment table (attachment table does not contain info about folder - this is everytime recalculated by script based on attachment ID)
- each 100 attachments are supposed to have new folder
- copy function creates new attach IDs in attachment table with reference to same physical file name (this is how it works in phpBB, files are not duplicated in real)
- if copied attachments have higher attach IDs then script is looking for them in new folder where files are not present in real.
wile
Registered User
Posts: 59
Joined: Mon Apr 30, 2007 4:21 pm

Re: Could anyone help with attachments subfolder MOD ?

Post by wile »

[ABD]Store phpBB3 attachments in subfolders
viewtopic.php?p=11929355#p11929355
- files are stored in folders based on attachment date
- DB contains info also about folder
- "copy topic" function does not work as DB is missing info about folder in attachment table for new items.

In some places UTF8 function was removed.. can this be issue in some cases ?

Code: Select all

#-----[ OPEN ]------------------------------------------
includes/functions_admin.php

#-----[ FIND ]------------------------------------------
		WHERE physical_filename = '" . $db->sql_escape(utf8_basename($filename)) . "'";

#-----[ REPLACE IT WITH ]------------------------------------------
		WHERE physical_filename = '" . $db->sql_escape($filename) . "'";
Otherwise I think it is quite nice solution. New folders should take approx. 50kB per year (12 folders).
wile
Registered User
Posts: 59
Joined: Mon Apr 30, 2007 4:21 pm

Re: Could anyone help with attachments subfolder MOD ?

Post by wile »

[DEV] Attach Mod
viewtopic.php?f=70&t=2144165
- also uses date for organising of folders
- when compared to other MODs there seem to be missing some code changes in specific files (I guess )
- not sure if my testing was wrong but files after removal in forum are not removed from folder in real
User avatar
Lumpy Burgertushie
Registered User
Posts: 69228
Joined: Mon May 02, 2005 3:11 am

Re: Could anyone help with attachments subfolder MOD ?

Post by Lumpy Burgertushie »

you should be addressing these issues in their respective topics.

however, the bottom line is that you need to get a better host that does not put this restrictive limit on you.

that will solve the problem without having to worry about any MODs that may or may not work.

the fact that none of the MODs you mention have been fixed or finished tells me that there is not enough need out there for it. that would be because there are very few hosts that impose these types of restrictions.


robert
Premium phpBB 3.3 Styles by PlanetStyles.net

I am pleased to announce that I have completed the first item on my bucket list. I have the bucket.
wile
Registered User
Posts: 59
Joined: Mon Apr 30, 2007 4:21 pm

Re: Could anyone help with attachments subfolder MOD ?

Post by wile »

wile wrote:[ABD]Store phpBB3 attachments in subfolders
viewtopic.php?p=11929355#p11929355
- files are stored in folders based on attachment date
- DB contains info also about folder
- "copy topic" function does not work as DB is missing info about folder in attachment table for new items.

In some places UTF8 function was removed.. can this be issue in some cases ?

Code: Select all

#-----[ OPEN ]------------------------------------------
includes/functions_admin.php

#-----[ FIND ]------------------------------------------
		WHERE physical_filename = '" . $db->sql_escape(utf8_basename($filename)) . "'";

#-----[ REPLACE IT WITH ]------------------------------------------
		WHERE physical_filename = '" . $db->sql_escape($filename) . "'";
Otherwise I think it is quite nice solution. New folders should take approx. 50kB per year (12 folders).
Hello,

So far there seems to be only one issue found related to copying of topic.
Fix should be following :
root/includes/mcp/mcp_main.php
find :

Code: Select all

'physical_filename'	=> (string) utf8_basename($attach_row['physical_filename']),
replace with :

Code: Select all

'physical_filename'	=> (string) ($attach_row['physical_filename']),
My question is for some technical people who know a lot about phpBB :-D

function utf8_basename filter out "/" or "\\".

Code: Select all

function utf8_basename($filename)
{
	// We always check for forward slash AND backward slash
	// because they could be mixed or "sneaked" in. ;)
	// You know, never trust user input...
	if (strpos($filename, '/') !== false)
	{
		$filename = utf8_substr($filename, utf8_strrpos($filename, '/') + 1);
	}

	if (strpos($filename, '\\') !== false)
	{
		$filename = utf8_substr($filename, utf8_strrpos($filename, '\\') + 1);
	}

	return $filename;
}
Do you think this solution is usable ? Or should we avoid removing of this check for user input.. for some security reasons ?
This solution uses adding also subfolder info into the DB and it would not be possible otherwise.

e.g.

Code: Select all

201312/someattachment.jpg -- function utf8_basename would remove first part with info about subfolder...
I tried file names with //// or \\\\ but does not seem to be possible to upload them into system anyway.
So from my perspective there is no security issue in this solution.
wile
Registered User
Posts: 59
Joined: Mon Apr 30, 2007 4:21 pm

Re: Could anyone help with attachments subfolder MOD ?

Post by wile »

Orphaned files tested
Here you are able to see orphaned files. This happens mostly if users are attaching files but not submitting the post. You are able to delete the files or attach them to existing posts. Attaching to posts requires a valid post ID, you have to determine this ID by yourself. This will assign the already uploaded attachment to the post you entered.
/acp/index.php?i=attachments&mode=orphan

Firstly I was a bit confused as this function seemed to be not working after implementing of this MOD. However I found a function that handles Orphaned attachments and this works well :

- handled by includes/acp/acp_attachments.php
- as stated in function // Just get the files with is_orphan set and older than 3 hours

Test case :
- set attachment time to be more than 3 hours ago - orphaned attachment was displayed in ACP
- tried to delete orphaned attachment (works well)
- tried to attach file to post number (works well)
wile
Registered User
Posts: 59
Joined: Mon Apr 30, 2007 4:21 pm

Re: Could anyone help with attachments subfolder MOD ?

Post by wile »

Who cares of attachments in subfolders

Results of investigation:
phpBB
- maybe planned in version 3.2

vbulletin
- attachments are organised per user ID, e.g. if user ID is 7724, then attachments will be in folder files/7/7/2/4/*
- basically each user has own folder, what is not definitive solution
- phpBB uses at least user ID as part of physical filename, eg. user 7724 uploaded attachment and it has name : 7724_71bda344c37df95c8a4ed8a3fa5e4e0

XENFORO
- Attachments are organised in sub-directories with a maximum of 1,000 per directory.
Movie
Registered User
Posts: 127
Joined: Sun Nov 02, 2014 2:54 pm
Location: Chicago Illinois Area.

Re: Could anyone help with attachments subfolder MOD ?

Post by Movie »

wile wrote:Who cares of attachments in subfolders

Results of investigation:
phpBB
- maybe planned in version 3.2

vbulletin
- attachments are organised per user ID, e.g. if user ID is 7724, then attachments will be in folder files/7/7/2/4/*
- basically each user has own folder, what is not definitive solution
- phpBB uses at least user ID as part of physical filename, eg. user 7724 uploaded attachment and it has name : 7724_71bda344c37df95c8a4ed8a3fa5e4e0

XENFORO
- Attachments are organized in sub-directories with a maximum of 1,000 per directory.
How do other social media sites who allow attachments manage attachments or does further investigation need to be taken.

I have been very aware of needing attachments spread over multiple folders for more than a decade back in the days when I host my own site from a home server managing the attachments was a breeze it was the same as managing and other file, backups were as simple as imaging the whole server hard drive, transferring files FTP was never an issue, no dealing with host tech people.

I would not want a new attachments folder created for each user who uploads a attachment while it would probably never present a problem on my site I worry about rather large phpbb sites with a lot of users, I rather see all attachments by any user stored in folders with a set amount of attachments per folder configurable in the APC so each phpbb site manager can select what works best for them.

Deleting a user and all their attachments or just deleting attachments in general is not a concern of mine meaning if I had my site configured to have 2000 attachments per folder then a year later deleted a few hundred attachments on my site having some folders with less than 2000 files is not a issue even if a situation presented itself and some attachment folders ended up with 100 files each so be it, I do not ever see that happening its not very often that attachments are removed from my phpbb site but considerations should be made for others.
View signatures turned off.
Movie
Registered User
Posts: 127
Joined: Sun Nov 02, 2014 2:54 pm
Location: Chicago Illinois Area.

Re: Could anyone help with attachments subfolder MOD ?

Post by Movie »

wile wrote:Who cares of attachments in subfolders

Results of investigation:
phpBB
- maybe planned in version 3.2

vbulletin
- attachments are organised per user ID, e.g. if user ID is 7724, then attachments will be in folder files/7/7/2/4/*
- basically each user has own folder, what is not definitive solution
- phpBB uses at least user ID as part of physical filename, eg. user 7724 uploaded attachment and it has name : 7724_71bda344c37df95c8a4ed8a3fa5e4e0

XENFORO
- Attachments are organised in sub-directories with a maximum of 1,000 per directory.
Add mybb to the list? http://community.mybb.com/thread-129800.html
Apparently mybb had thought about of large amounts of files in a single folder or maybe it was a accident?
View signatures turned off.

Return to “[3.0.x] MOD Writers Discussion”