Storing attachments

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Suggested Hosts
Locked
Budovi
Registered User
Posts: 10
Joined: Sun Nov 21, 2010 8:01 pm

Storing attachments

Post by Budovi »

Hi all,

attachments stored with phpbb3 has no prefix. Because of that, I lost all my files from two forums.

Hmm stupid mistake, but I did not realize that files with no prefix are transferred by default using ASCII mode in some FTP clients (used FileZilla), so this messed up all my files. Now they are completely corrupted with no chance to repair (maybe some archives with repair possibility)

Of course I always should check it, but automatic transfer mode is just comfort and I did not realized this mistake possibility.

But idea of adding some prefix (maybe *.file, or randomly generated with check that is not any ASCII files prefix) can prevent this stupid mistake, I believe there were more people with this problem...
Pony99CA
Registered User
Posts: 4783
Joined: Thu Sep 30, 2004 3:13 pm
Location: Hollister, CA
Name: Steve
Contact:

Re: Storing attachments

Post by Pony99CA »

First, that's not a "prefix", that's an extension. The "phpbb_" in the table name phpbb_users is a prefix.

Second, did you try uploading your files again as ASCII and then downloading them as binary? That might fix things (if my limited understanding of FTP transfers is correct).

Steve
Silicon Valley Pocket PC (http://www.svpocketpc.com)
Creator of manage_bots and spoof_user (ask me)
Need hosting for a small forum with full cPanel & MySQL access? Contact me or PM me.
Budovi
Registered User
Posts: 10
Joined: Sun Nov 21, 2010 8:01 pm

Re: Storing attachments

Post by Budovi »

OMG of course I'm talking about file types (file extensions) :oops:
not sure what I was thinking about when writing....

no, recovery is not possible, in ASCII mode file is "converted" between unix-style line ending ("\n") and ms-dos (of course "ms-dos" and windows still running on this crap must be different) line ending ("\r\n")

if you upload to server all \r\n will be overwritten with \n, if you download it turns back \n to \r\n

but when you open binary file in ASCII text editor there can be these "strings" too, they are overwritten, trying upload and download again can mess up it more than they are, depending on file,,

as I said, no chance to recover it, I think these file should have some (safe) extension
User avatar
bantu
3.0 Release Manager
3.0 Release Manager
Posts: 2523
Joined: Mon Jul 10, 2006 9:58 pm
Name: Andreas Fischer

Re: Storing attachments

Post by bantu »

Your issue, as bad it is and as sorry I feel for you, is clearly an issue with FileZilla's unsafe default settings and I personally have reported this to the authors of FileZilla at http://trac.filezilla-project.org/ticket/4235 14 months ago.

Deciding whether or not to use ASCII transfer mode based on file extension is a very very bad idea from the start. Transfering everything "as is" (i.e. in BINARY transfer mode) is exactly what FTP should be all about today. Because in the end FTP stands for "File Transfer Protocol" and not "File Transfer and Change Random Bits Protocol".

Also, even if attachment files had an extension in phpBB, that would not guarantee anything in general and a broken program could still screw up and throw away random bits (either by using ASCII mode on binary files or because of a programming error) because FTP also does not verify whether the file you got is the same as the file on the FTP server. If you want to verify that, you should use a more advanced protocol such as rsync with the checksum option (which however will probably only be available in more expensive hosting packages).
Powered by Coffee
User avatar
Lumpy Burgertushie
Registered User
Posts: 69223
Joined: Mon May 02, 2005 3:11 am
Contact:

Re: Storing attachments

Post by Lumpy Burgertushie »

none of that means anything to 99% of the users out there in the world.

the truth is that phpbb should never have adopted this way of storing the files.

it makes no sense at all to remove the extensions from them. any ftp program set on auto will do the same thing. as far as I know, the auto setting in any ftp program looks at file extensions to make it's choice of whether to download in ascii or binary. that is the standard and has been forever.

and contrary to what some believe, binary is not good for all files any more than ascii is .

binary should be reserved for images, and exe and zip etc. all text based files like php, html,cfg, etc.
should be transferred in ascii.

as we have all seen over the years, if a file is transferred in the wrong form then it can be corrupted in the process.

as we have seen here since the change was made to remove all extensions from attachments, it corrupts the image and other types of files almost 100% of the time. the only reason we don't hear about it more often is because most people never have the need to restore that backup they made of those attachments so they never know that they are no longer any good.


oh well, there I go again;

Image

robert

EDIT: After having read through that bug report to filezilla, I do agree that it should be set to "as is" by default.

however, since it is not and I doubt that many ftp programs even have that option, I still think the files should have the extensions left on them.

robert
Premium phpBB 3.3 Styles by PlanetStyles.net

I am pleased to announce that I have completed the first item on my bucket list. I have the bucket.
Budovi
Registered User
Posts: 10
Joined: Sun Nov 21, 2010 8:01 pm

Re: Storing attachments

Post by Budovi »

Hmm agree with Lumpy Burgertushie,,

today "automatic transfer mode" is providing by almost any FTP client,
just a little inattention and all files are gone.

This problem HAS solution with adding extension, as I said *.file, *.pstf, ... or something that is not in list for ASCII transfer and I think this should be added in next phpbb relase.

I don't think this is bug of clients, just default setting, maybe it should ask right before uploading / downloading extensionless file... It is better for FTP client to ask than has it defined...
This opinion still did not change my point of view, phpbb should store files with safe extension.

I had to reconcile with the loss of files (it was shock), this thread is not "for information" or about recovery, this thread is about how to save other users from this, it is about ideas for fix

This will never happen to me again :roll: but I try to take care for others...
Pony99CA
Registered User
Posts: 4783
Joined: Thu Sep 30, 2004 3:13 pm
Location: Hollister, CA
Name: Steve
Contact:

Re: Storing attachments

Post by Pony99CA »

Budovi wrote:no, recovery is not possible, in ASCII mode file is "converted" between unix-style line ending ("\n") and ms-dos (of course "ms-dos" and windows still running on this crap must be different) line ending ("\r\n")

if you upload to server all \r\n will be overwritten with \n, if you download it turns back \n to \r\n
Yes, but I said upload as ASCII and download again as binary. I'm hoping that uploading as ASCII will undo the damage and then you can download everything as binary and it will be a good backup. (If you want to actually edit any text-like files, that's another matter.)

Of course, if DOS files still have an end-of-file marker -- they used Ctrl+Z back in the day -- and your binary file contained that, that ASCII file may have been truncated and be unrecoverable.

Have you actually tried it? If not, what have you got to lose?

Steve
Silicon Valley Pocket PC (http://www.svpocketpc.com)
Creator of manage_bots and spoof_user (ask me)
Need hosting for a small forum with full cPanel & MySQL access? Contact me or PM me.
Budovi
Registered User
Posts: 10
Joined: Sun Nov 21, 2010 8:01 pm

Re: Storing attachments

Post by Budovi »

Of course I tried it, bud realize that "line ending characters" are probably in every (binary) file, it would be great luck if they did not.... Binary files contains lots of data and there is big chance that this little string is there also...
User avatar
Sajaki
Registered User
Posts: 1390
Joined: Mon Mar 02, 2009 1:41 pm
Location: Amsterdam
Contact:

Re: Storing attachments

Post by Sajaki »

And after 10 years we still have this issue.
User avatar
HiFiKabin
Community Team Member
Community Team Member
Posts: 6671
Joined: Wed May 14, 2014 9:10 am
Location: Swearing at the PC, UK
Name: James
Contact:

Re: Storing attachments

Post by HiFiKabin »

Sajaki wrote: Sun Apr 18, 2021 7:15 am
Please check the date before replying to a topic. This is 10 years old, hence closing
User avatar
stevemaury
Support Team Member
Support Team Member
Posts: 52768
Joined: Thu Nov 02, 2006 12:21 am
Location: The U.P.
Name: Steve
Contact:

Re: Storing attachments

Post by stevemaury »

Sajaki's point was, after all this talk and 10 years, nothing has been done about this. But I question his use of "we", as it is not really a phpBB issue. There are lot of reasons not to use file extensions for phpBB attachments.
I can stop all your spam. I can upgrade or update your Board. PM or email me. (Paid support)
Locked

Return to “phpBB Discussion”