Page 1 of 2

Uploading attachments issues

Posted: Fri Mar 22, 2013 9:58 am
by DoYouSpeakWak
Support Request Template
What version of phpBB are you using? phpBB 3.0.11
What is your board's URL? http://www.nordisk-forum.dk
Who do you host your board with? fusioned.net
How did you install your board? I used the download package from phpBB.com
What is the most recent action performed on your board? Update from a previous version of phpBB3
Is registration required to reproduce this issue? No
Do you have any MODs installed? Yes
What version of phpBB3 did you update from? phpBB 3.0.10
What MODs do you have installed? https://www.phpbb.com/customise/db/mod/thanks_for_posts
https://www.phpbb.com/customise/db/mod/nv_who_was_here
https://www.phpbb.com/customise/db/mod/prime_links
Tapatalk
What styles do you currently have installed? subsilver2_new_colors
What language(s) is your board currently using? Danish
Which database type/version are you using? MySQL 5
What is your level of experience? Experienced with PHP and phpBB
When did your problem begin? After the move to a new host
Please describe your problem. 1: On the old hosts, all the attachments works fine.

2: moved to new hoss. 9998 attachments downloaed and uploaded in binary mode.
rest normally. database extracted via phpmyadmin.

3: uploaded both to a local lamp setup and tested everythig prior to remote upload. Both setups seems to work just fine. exepct that only about half of the attachments (all images) does not show.resyncing the attachments in stk didnt do any difference.

Example of problem:

http://www.nordisk-forum.dk/viewtopic.php?f=49&t=43847

Short: only half of images (attachments) show up after a move. triple checked that it was done by the book. What should i try next ?
Generated by SRT Generator

Re: Uploading attachments issues

Posted: Fri Mar 22, 2013 11:17 am
by Oyabun1
You could use the script in this article, Backing up attachments with their original filenames, to backup the files and then you could see if the images open directly from the server and when downloaded. That should give an indication of whether the problems is with the files.

Re: Uploading attachments issues

Posted: Fri Mar 22, 2013 12:07 pm
by DoYouSpeakWak
Thx. here is how it went.

This was run on my localhost with a copy of the board. Ran rename.php

It told me that there was 30000+ attachments to fetch, but there is only 9998 files in /files/. Mmm onwards. changed the interval to 1500. when done i had 9998 attachments in filebackup all read able. So thats good.

Now for the questions:

1: All the files are valid, no corrupt after this script extracted them. But why did it belive there was 30K images and not 10K ?

2: What now ? The attachments on the board is still not showing in most post. about 50-60%. anyway to fix this besides adding each image manually to the posts again ?

Re: Uploading attachments issues

Posted: Fri Mar 22, 2013 12:38 pm
by Oyabun1
DoYouSpeakWak wrote:Only very few files in /files/ are non cryptical in name. most are just called TSGR35RETE45364 etc. no names.jpg or anything.
All the files in there should have hashed file names and no file extensions (except for .htaccess and index.htm). It's a security feature, some of the reasons are covered in this Blog article.

If the files were transferred in binary mode it is strange that they have become corrupted.

Re: Uploading attachments issues

Posted: Fri Mar 22, 2013 12:51 pm
by DoYouSpeakWak
Indeed. I can confirm that the extracted images are 100% match and valid. found 25 images that shows on the forum. copied them to HD and compared that dir with filebackup. all 25 had succesfully been extracted.

In files/ there is about 30 pics with a .jpg extension, rest is standard encoded.

Whats strange is that about half actually show fine after the transfer of the site. During which no files were changed and the db wasnt touched in anyway. not even the config table.

These guys are serious about their attachments, DIY forums etc going years back. I have redownloaded the /files/ files 4 times now in binary, reupped them locally and remotely. gone of the proccess many times on local. all the time its the same results.

Only part of this move that im not 100% about is the db extraction. It was done via phpmyadmin and the guide here

https://www.phpbb.com/kb/article/transf ... or-domain/

was followed. Since the org host had a older version of phpmyadmin i had to do this as the guide said
In the right-hand frame, select the Export tab. In the Export section of this page, be sure to choose Select All as well as the SQL radio button. In the Options section of this page, ensure the Add DROP TABLE checkbox is checked.

Below this setting in the Data section, ensure Complete inserts is checked, and Extended inserts is unchecked.
I had to stray a bit on that. If i didnt include "Extended inserts" the database would only be restored with 50 tables (tried it 3 times) when included "Extended inserts" then it worked fie. All other steps of the guide was followed besides this one. And every aspect of the board functions fine. (besides permission tracing that gives a module not know, but that shouldnt matter)

Re: Uploading attachments issues

Posted: Fri Mar 22, 2013 1:40 pm
by T0ny
How many rows are there in your phpbb_attachments table?

Re: Uploading attachments issues

Posted: Fri Mar 22, 2013 1:47 pm
by Brf
DoYouSpeakWak wrote: It told me that there was 30000+ attachments to fetch, but there is only 9998 files in /files/.
Likely, whatever software you, or your host, were using to access the folder ran into a file-limit.

Re: Uploading attachments issues

Posted: Fri Mar 22, 2013 1:54 pm
by DoYouSpeakWak
T0ny wrote:How many rows are there in your phpbb_attachments table?
33,991

acp. number of attachments. 33,980

I didnt even think that there could be more than 9998 since thats whats in files.
Brf wrote:
DoYouSpeakWak wrote: It told me that there was 30000+ attachments to fetch, but there is only 9998 files in /files/.
Likely, whatever software you, or your host, were using to access the folder ran into a file-limit.
that would mean that my ftp program, filezilla. when loading /files/ are limited to 9998 files ?

Edit: seems to be a server issue. filezilla has no such limits.
http://forum.filezilla-project.org/view ... f=1#p58014

Image

Re: Uploading attachments issues

Posted: Fri Mar 22, 2013 2:04 pm
by T0ny
I had a problem a few years ago where I could only see 1,998 files in a directory by FTP. It turned out that the host's FTP server was limited to showing 2,000 files ('.' & '..' being the other 2). I had to download the files that I could see, then delete them so the others would become visible.

Maybe your hosts FTP server is limited to 10,000 ?

Re: Uploading attachments issues

Posted: Fri Mar 22, 2013 2:07 pm
by DoYouSpeakWak
T0ny wrote:I had a problem a few years ago where I could only see 1,998 files in a directory by FTP. It turned out that the host's FTP server was limited to showing 2,000 files ('.' & '..' being the other 2). I had to download the files that I could see, then delete them so the others would become visible.

Maybe your hosts FTP server is limited to 10,000 ?
Maybe. that would be an option but i hate to delete them from the only place were they are backed up. The move was done due to the old host only pay bills. no support. Couldnt even get ssh access to do a mysql dump.

But i have contacted the old host again. hopefull that they can change the limits or give me ssh access. (its a dedicated)

Would be nice to confirm that this really is the issue, even though it looks like it.

Re: Uploading attachments issues

Posted: Fri Mar 22, 2013 2:46 pm
by Brf
from what I am reading, Filezilla itself does not have a limit, but your hosting might limit folder listings to 10000.

Re: Uploading attachments issues

Posted: Fri Mar 22, 2013 3:43 pm
by DoYouSpeakWak
Brf wrote:from what I am reading, Filezilla itself does not have a limit, but your hosting might limit folder listings to 10000.
I agree. i have contacted him. he refuses to give me ssh access for 2 hours or change the ftp settings. but did offer to zip that directory and put it up for download to me.

But that worries med with binary mode. If he goes in via ssh and zip the directory and makes it readable so i can download it. Then im not sure the files will be in the right mode even if i download the zip in binary. Any ideas on that ?

Re: Uploading attachments issues

Posted: Fri Mar 22, 2013 3:50 pm
by Brf
A zip file, by definition, is a binary file. If the folder is zipped on the server, it would contain the files in binary.

Re: Uploading attachments issues

Posted: Fri Mar 22, 2013 3:53 pm
by stevemaury
All you can do is try it.

Re: Uploading attachments issues

Posted: Fri Mar 22, 2013 3:54 pm
by DoYouSpeakWak
Thx very much. what i though. should be fine as long as he doesn use ssh, ftp to transfer them before zipping. direct in the terminal should work fine.

To be honest its the first time i have encountered such a limit on a server. In the future i will always check amount of files, and those listed in acp and the database before starting the transfer. And not just think that the number shown in /files/ are correct.