Old uploads broken after migration

Get help with installation and running phpBB 3.2.x here. Please do not post bug reports, feature requests, or extension related questions here.
Post Reply
mixiekins
Registered User
Posts: 2
Joined: Mon Mar 18, 2019 3:24 am

Old uploads broken after migration

Post by mixiekins »

I want to mention first that I am aware that I need to update to the newest version of pbpBB but I'm concerned that this issue has to be resolved beforehand, as such a big change could make matters worse. I'm happy to do the update first if need be, but I'm far from any authority on phpBB and should do my due diligence to ask ahead of time in case I should try something before backing up and doing the upgrade.


Support Request Template
What version of phpBB are you using? phpBB 3.1.12
What is your board's URL? https://www.pokemon-mini.net/forum
Who do you host your board with? prgmr.com
How did you install your board? I used the download package from phpBB.com
What is the most recent action performed on your board? Fresh Install
Is registration required to reproduce this issue? Yes
Do you have any MODs installed? No
Do you have any extensions installed? No
What styles do you currently have installed? prosilver
What language(s) is your board currently using? English
Which database type/version are you using? MariaDB
What is your level of experience? Comfortable with PHP and phpBB
What username can be used to view this issue? No answer given
What password can be used to view this issue? No answer given
What actions did you take (updating your board; installing a MOD, style or extension; etc.) prior to this problem becoming noticeable? I migrated the site from one hosting company to another; I took over for the previous admin. All of the file attachments for posts which predate the migration are corrupted.
Please describe your problem. The migration was performed with a .7z generated on the origin server pulled to the destination server via wget and went smoothly; the previous admin did not provide a checksum, so to rule out an error in the .7z compression/extraction I've copied over the files themselves (namely those which are in the directory named "files" which is found within the forum's root directory) anew as a .zip (via wget as well). This was unzipped within the destination server once received, and replaced the malfunctioning "files" directory.

I have cleared my cache and used a browser that has never viewed the test pages, but the issue persists; in fact, the images for posts which have been made post-migration should now be broken, but they still work, so I wonder if there is some phpbb cache I need to clear. I'm familiar with server administration and have installed phpbb plenty of times, but I'm less familiar with the inner workings of this BBS. It was suggested that, if using FTP, this could happen if the files are not uploaded with binary mode, but this was pulled to my server via wget, so I'm not sure if the point is moot or not.

The migration was performed on Feb 23rd of this year, so I've tested uploads by going into the ACP > Posting > Manage attachments > and then selected image files that predate the move, and opening them in another tab.

They will either show up garbled and corrupted example:
8d52e1e3ly1g09vu4hp8yj20sg0lcal9.jpg
February 17th, 2019, 11:27
https://www.pokemon-mini.net/forum/down ... e37452ebd0

Or, it will display the text which reads: "The image "[URL HERE]" cannot be displayed because it contains errors.
example:
ditto.png
January 26th, 2019, 19:58
https://www.pokemon-mini.net/forum/down ... e37452ebd0

And in-thread they show up as blank little boxes:
https://www.pokemon-mini.net/forum/view ... 1058#p2635

However, all uploads that are post-migration work fine:
test.jpg
March 15th, 2019, 19:46
https://www.pokemon-mini.net/forum/down ... e37452ebd0
which can be seen working on:
https://www.pokemon-mini.net/forum/view ... ebd0#p6646

A user login isn't needed to see the issue, and while you can log in to upload (which would work, as it's post-migration) I can't generate more malfunctioning files since they came from the old server and would require time traveling. :) Never the less, I can set up a regular user if this helps at all.

I have also attempted deletion of orphaned attachments and renewed my "file statistics" (there was a big red warning at ACP > Posting > Manage attachments) but no dice.
Generated by SRT Generator
.m.
Registered User
Posts: 539
Joined: Wed Nov 04, 2009 8:39 pm

Re: Old uploads broken after migration

Post by .m. »

can you check whether original files are working fine in an image viewer ?

[eg. drag a file from the backup's files folder to viewer like irfanview]
User avatar
Mick
Support Team Member
Support Team Member
Posts: 22743
Joined: Fri Aug 29, 2008 9:49 am
Location: Watching cricket probably.

Re: Old uploads broken after migration

Post by Mick »

Maybe this Knowledge Base - Transferring attachment files with Filezilla and you need to check your cookie settings Knowledge Base - Fixing incorrect cookie settings - the Cookie domain should be .pokemon-mini.net, note the leading dot, and cookie secure should be enabled if you're using SSL.
"The more connected we get the more alone we become" - Kyle Broflovski©
User avatar
david63
Registered User
Posts: 18303
Joined: Thu Dec 19, 2002 8:08 am
Location: Lancashire, UK
Contact:

Re: Old uploads broken after migration

Post by david63 »

Unzipping files on a server is not recommended as the directory structure is not always maintained. Try unzipping locally and then uploading the unzipped files.
David
Remember: You only know what you know and - you don't know what you don't know!
My CDB Contributions | How to install an extension
I will not be accepting translations for any of my extensions in Github - please post any translations in the appropriate topic.
No support requests via PM or email as they will be ignored
User avatar
thecoalman
Community Team Member
Community Team Member
Posts: 4061
Joined: Wed Dec 22, 2004 3:52 am
Location: Pennsylvania, U.S.A.
Contact:

Re: Old uploads broken after migration

Post by thecoalman »

Check the permissions and owner of the files and the owner/permissions for the files folder itself, the files should be the same as the test file that worked. The files folder should be the same as if you created new folder via ftp and the permissions should be set so it's writable by phpBB. You should also check the server error logs as it may give you an indication of what the problem is.

All attachment files should be in "files" and there is only two additional files. an .htacces file and index.htm.
“Results! Why, man, I have gotten a lot of results! I have found several thousand things that won’t work.”

Attributed - Thomas Edison
User avatar
EA117
Registered User
Posts: 1805
Joined: Wed Aug 15, 2018 3:23 am
Contact:

Re: Old uploads broken after migration

Post by EA117 »

mixiekins wrote:
Mon Mar 18, 2019 4:04 am
They will either show up garbled and corrupted example:
8d52e1e3ly1g09vu4hp8yj20sg0lcal9.jpg
February 17th, 2019, 11:27
https://www.pokemon-mini.net/forum/down ... e37452ebd0
That .JPG file's bytes contains a suspiciously high number of 0x0D 0x0A sequences, which makes me suspect the issue behind why Mick linked you to Transferring attachment files with Filezilla. Meaning the files were transferred as ASCII or Auto instead of explicitly as Binary by your FTP program. (Which I assume in this case is WGET, and not that you were using WGET in HTTP mode during any portion of this process.)

That's actually inconsistent with the fact that you're describing having used server-size .ZIP/.7z compression and decompression during some of the attempts. Although there are other issues for which avoiding the server-side compression or extraction is usually recommended, "will translate your binary files as though they are text" isn't one of the expected issues. (Or at least not one I've seen mentioned.) So I'm guessing this corrupted example was actually transferred by FTP at some point, as an individual file.

It's not that the 0x0D 0x0A (CR/LF) sequence "cannot" occur naturally within an image's bytes. But there isn't a single 0x0A byte I've found within this example image which isn't part of a 0x0D 0x0A pair. When there are this many instances, it's usually because the FTP Auto or ASCII mode was translating every natural instance of 0x0A in the image's bytes into 0x0D 0x0A, hence corrupting the image (or whatever type of binary file it originally was).

Edit: Just realized I should have pointed out: the direction of "0x0A became 0x0D 0x0A" is actually indicating the file became corrupted during download from the presumably-Linux server. Which is when translation from Linux (LF) to Windows (CR/LF) would have occurred in response to Auto or ASCII mode. The file may in fact have been "correctly" uploaded to the new server, but was already corrupt at that point.

Bottom line: In addition to ruling out the other suggestions, can you re-download from the original server in binary mode, and then re-upload at least the files folder in binary mode. If this is your issue, note any other files you have brought from the old site (styles, images, etc.) can also be impacted by this issue. And attachments is where you're noticing it "first".
User avatar
stevemaury
Support Team Member
Support Team Member
Posts: 51547
Joined: Thu Nov 02, 2006 12:21 am
Location: The U.P.
Name: Steve
Contact:

Re: Old uploads broken after migration

Post by stevemaury »

First make sure that your FTP client is set such that all files are transferred binary, without regard to their extension or lack thereof. If you are using Filezilla, see this - https://www.phpbb.com/support/docs/en/3 ... filezilla/ .

Then go to the origin server, download the /files folder to your local computer, rename the files folder on the destination server and upload the /files folder on your local computer to the destination server.
For REALLY good and VERY inexpensive hosting CLICK HERE

I can stop all your spam. I can upgrade or update your Board. PM or email me. (Paid support)
mixiekins
Registered User
Posts: 2
Joined: Mon Mar 18, 2019 3:24 am

Re: Old uploads broken after migration

Post by mixiekins »

.m. wrote:
Mon Mar 18, 2019 7:38 am
can you check whether original files are working fine in an image viewer ?
[eg. drag a file from the backup's files folder to viewer like irfanview]
irfanview.JPG
for reference, they are within the "files" folder:
files-folder.JPG
Mick wrote:
Mon Mar 18, 2019 7:41 am
Maybe this Knowledge Base - Transferring attachment files with Filezilla and you need to check your cookie settings Knowledge Base - Fixing incorrect cookie settings - the Cookie domain should be .pokemon-mini.net, note the leading dot, and cookie secure should be enabled if you're using SSL.
I'm using wget, not Filezilla. It is a single .zip file that is pulled and extracted within the server.
I've tried the change you suggested, it was set as follows:
cookie-with-leading-dot_before.JPG
and now is set like so:
cookie-with-leading-dot_after.JPG
but the issue persists-- pre-migration attachments are corrupted, post-migration are working fine.
I'm curious, if cookies was the issue, would it not have prevented the new attachments from working as well? To recap, the attachments files brought from the old server are the corrupted ones, while newly-posted ones are functioning correctly.
david63 wrote:
Mon Mar 18, 2019 8:55 am
Unzipping files on a server is not recommended as the directory structure is not always maintained. Try unzipping locally and then uploading the unzipped files.
I did indeed try this as well, to no avail. (Further, the files are showing as corrupted when viewed in irfanview as mentioned above. (Yes, when using FileZilla for individual files in the way, I did try binary mode, per Mick's KB link, it was the first thing I tried when encountering the issue.)
thecoalman wrote:
Mon Mar 18, 2019 1:12 pm
Check the permissions and owner of the files and the owner/permissions for the files folder itself, the files should be the same as the test file that worked. The files folder should be the same as if you created new folder via ftp and the permissions should be set so it's writable by phpBB. You should also check the server error logs as it may give you an indication of what the problem is.

All attachment files should be in "files" and there is only two additional files. an .htacces file and index.htm.
Already did that.
stevemaury wrote:
Mon Mar 18, 2019 2:40 pm
First make sure that your FTP client is set such that all files are transferred binary, without regard to their extension or lack thereof. If you are using Filezilla, see this - https://www.phpbb.com/support/docs/en/3 ... filezilla/ .

Then go to the origin server, download the /files folder to your local computer, rename the files folder on the destination server and upload the /files folder on your local computer to the destination server.
Thanks for trying to help, but that's already been covered.
EA117 wrote:
Mon Mar 18, 2019 1:48 pm
mixiekins wrote:
Mon Mar 18, 2019 4:04 am
They will either show up garbled and corrupted example:
8d52e1e3ly1g09vu4hp8yj20sg0lcal9.jpg
February 17th, 2019, 11:27
https://www.pokemon-mini.net/forum/down ... e37452ebd0
That .JPG file's bytes contains a suspiciously high number of 0x0D 0x0A sequences, which makes me suspect the issue behind why Mick linked you to Transferring attachment files with Filezilla. Meaning the files were transferred as ASCII or Auto instead of explicitly as Binary by your FTP program. (Which I assume in this case is WGET, and not that you were using WGET in HTTP mode during any portion of this process.)

That's actually inconsistent with the fact that you're describing having used server-size .ZIP/.7z compression and decompression during some of the attempts. Although there are other issues for which avoiding the server-side compression or extraction is usually recommended, "will translate your binary files as though they are text" isn't one of the expected issues. (Or at least not one I've seen mentioned.) So I'm guessing this corrupted example was actually transferred by FTP at some point, as an individual file.

It's not that the 0x0D 0x0A (CR/LF) sequence "cannot" occur naturally within an image's bytes. But there isn't a single 0x0A byte I've found within this example image which isn't part of a 0x0D 0x0A pair. When there are this many instances, it's usually because the FTP Auto or ASCII mode was translating every natural instance of 0x0A in the image's bytes into 0x0D 0x0A, hence corrupting the image (or whatever type of binary file it originally was).

Edit: Just realized I should have pointed out: the direction of "0x0A became 0x0D 0x0A" is actually indicating the file became corrupted during download from the presumably-Linux server. Which is when translation from Linux (LF) to Windows (CR/LF) would have occurred in response to Auto or ASCII mode. The file may in fact have been "correctly" uploaded to the new server, but was already corrupt at that point.

Bottom line: In addition to ruling out the other suggestions, can you re-download from the original server in binary mode, and then re-upload at least the files folder in binary mode. If this is your issue, note any other files you have brought from the old site (styles, images, etc.) can also be impacted by this issue. And attachments is where you're noticing it "first".
Ahh, now we're on to something. Copy that, I'm seeing the same. I tried viewing them on a Linux workstation (more forgiving about this sort of thing, moreso than even irfanview), and, for the most part, they're working--
linux-view.jpeg
and upon closer inspection it does look like there's an issue with the line termination. I'm thinking now that the admin of the old server had been using Filezilla to download to his pc (which would be Windows), compress it, then upload that botched set of files for me to wget. I really wish that I could have been given access to the old server to just server-to-server rsync to avoid all of this, but I've not been given access, regrettably. :(

Well, tried a diff of the extraction of the .zip on that linux workstation, and interestingly enough, they were actually different from what was on the server, so for whatever reason, the extraction on windows and serverside (odd, as the server is Centos) were both getting corrupted.
diff-of-linux-extract.jpeg
That's been uploaded with a more reliable .tar.gz and all is well with the world.

TL;DR: EA117 is legit, and for anyone with this problem, please just do server-to-server rsync if you can be granted access to the origin server, or extract on debian if you're using something other than .tar.gz if you're stuck in the same boat I was.

Now I can continue with upgrading. Good to know that this won't get in the way, phew!
Thanks for the help, let's mark this one solved.
User avatar
thecoalman
Community Team Member
Community Team Member
Posts: 4061
Joined: Wed Dec 22, 2004 3:52 am
Location: Pennsylvania, U.S.A.
Contact:

Re: Old uploads broken after migration

Post by thecoalman »

Glad it worked out for you.
mixiekins wrote:
Sat Mar 23, 2019 10:18 pm
for reference, they are within the "files" folder:
Quick note here. It shouldn't make a difference but there should be no folder named plupload in the files folder. Other than the files with the random string there should only* be an .htaccess file and index.htm. *If the forum was originally phpBB2 with attachment mod you will have files with name and extension that were uploaded prior to upgrading to phpBB3.
Last edited by thecoalman on Sun Mar 24, 2019 1:19 am, edited 1 time in total.
Reason: Mistake, see below.
“Results! Why, man, I have gotten a lot of results! I have found several thousand things that won’t work.”

Attributed - Thomas Edison
User avatar
3Di
Former Team Member
Posts: 16032
Joined: Mon Apr 04, 2005 11:09 pm
Location: Milan (IT) Frankfurt (DE)
Name: Marco
Contact:

Re: Old uploads broken after migration

Post by 3Di »

thecoalman wrote:
Sun Mar 24, 2019 12:55 am
there should be no folder named plupload in the files folder
Such folder is created AFAIK during the first attachment's upload.
Isn't present into a vanilla download package.
Attachments
attach_plupload.png
attach_plupload.png (26.5 KiB) Viewed 504 times
Please PM me only to request paid works. Thx.
Want to compensate me for my interest? Donate
My development's activity º PhpStorm's proud user
Extensions, Scripts, MOD porting, Update/Upgrades
Looking for a specific feature or alternative option? We will rock you! 🚀
User avatar
thecoalman
Community Team Member
Community Team Member
Posts: 4061
Joined: Wed Dec 22, 2004 3:52 am
Location: Pennsylvania, U.S.A.
Contact:

Re: Old uploads broken after migration

Post by thecoalman »

3Di wrote:
Sun Mar 24, 2019 1:06 am
thecoalman wrote:
Sun Mar 24, 2019 12:55 am
there should be no folder named plupload in the files folder
Such folder is created AFAIK during the first attachment's upload.
Isn't present into a vanilla download package.
You are correct, my mistake.
“Results! Why, man, I have gotten a lot of results! I have found several thousand things that won’t work.”

Attributed - Thomas Edison
.m.
Registered User
Posts: 539
Joined: Wed Nov 04, 2009 8:39 pm

Re: Old uploads broken after migration

Post by .m. »

irfanview saying -- unknown file format / damaged file
that might mean that all such files cannot be used without repair :(
User avatar
canonknipser
Registered User
Posts: 2096
Joined: Thu Sep 08, 2011 4:16 am
Location: Germany
Name: Frank Jakobs
Contact:

Re: Old uploads broken after migration

Post by canonknipser »

.m. wrote:
Sun Mar 24, 2019 2:53 am
cannot be used without repair
There are reports of users who could repair a majority of those broken images. See this topic at the german support board (you may want to translate it with your favourite online translation tool) for a happy user ;)
Greetings, Frank
phpbb.de support team member
English is not my native language - no support via PM or mail
New arrival - Extensions and scripts for phpBB
Post Reply

Return to “[3.2.x] Support Forum”