Done 3.2.2 to 3.2.7 upgrade but inline attachments missing

Get help with installation and running phpBB 3.2.x here. Please do not post bug reports, feature requests, or extension related questions here.
User avatar
EA117
Registered User
Posts: 1037
Joined: Wed Aug 15, 2018 3:23 am
Contact:

Re: Done 3.2.2 to 3.2.7 upgrade but inline attachments missing

Post by EA117 » Wed Jul 24, 2019 3:47 am

For what it's worth, I'm still seeing consistently 404 for the failed attachments, and never 500. Which is "a good thing" to me, since 500 implies a more serious error. But still not obvious why the 404 is happening. I see there is a 404 happening on the avatars in some of those example links, too.

The next thing I would recommend is still the previous thing I would recommend:
EA117 wrote:
Tue Jul 16, 2019 3:04 pm
So to take a specific example, file.php?id=4095 again, you saw in the database what the "physical filename" value was set to. And you went to the /files directory and were able to find that specific physical filename. And that file was not zero length. And copying that file and renaming it to the correct original image name shows a valid and non-corrupt image.

What were the permissions on that specific file? (e.g. 666, 644, 777, etc.) Is the alphabetic case of the "physical filename" value in the database an exact match? (e.g. Not "2_47ca94c7e0d7f7dd631a1a1b0b45c296" versus "2_47CA94c7e0D7f7dd631a1a1B0B45C296" or similar.)

Again, we're just looking for any reason or excuse that "asking phpBB about attachment ID 4095" might not have been able to find or use the file described in the database for attachment ID 4095.
The one additional thing I would add now is "maybe it's just having a hard time finding the thumbnail image, instead of the actual attachment?"

So when looking at the phpbb_attachments table entry for attachment ID 4095, can you also confirm whether the field "thumbnail" is set to "0" or "1"?

If "1", in addition to the "2_47ca94c7e0d7f7dd631a1a1b0b45c296" you were testing exists, and ensuring was valid, and checking the permissions, also look for a file named "thumb_2_47ca94c7e0d7f7dd631a1a1b0b45c296". And do all the same checks and reports regarding that file as well.

robbell
Registered User
Posts: 20
Joined: Sun Feb 27, 2011 12:33 am

Re: Done 3.2.2 to 3.2.7 upgrade but inline attachments missing

Post by robbell » Tue Jul 30, 2019 11:32 am

I've been into cpanel to match up the attachments, and get the following -

4095 has the physical id 10453_68e0bde9cd5e0f9160859dc175ed87e6 which is not present in the files folder. Thumbnail is set to 1
4096 has the physical id 10453_c63365e903ca51d8bb64734746f6d668 which is also not present in the files folder. Thumbnail is set to 1

Where they could have gone, I have no idea, I've never deleted anything from the files folder...

I have noticed there's an in_message field, everything is set to 0 in that field - could that be indicating a problem?

This is all way over my head I must confess.

User avatar
EA117
Registered User
Posts: 1037
Joined: Wed Aug 15, 2018 3:23 am
Contact:

Re: Done 3.2.2 to 3.2.7 upgrade but inline attachments missing

Post by EA117 » Tue Jul 30, 2019 4:25 pm

robbell wrote:
Tue Jul 30, 2019 11:32 am
4095 has the physical id 10453_68e0bde9cd5e0f9160859dc175ed87e6 which is not present in the files folder. Thumbnail is set to 1
4096 has the physical id 10453_c63365e903ca51d8bb64734746f6d668 which is also not present in the files folder. Thumbnail is set to 1
Well, "good news" in a way, then. Since that's a perfect explanation for why viewing the discussion topic shows 404 and broken images for all of these attachments.

Just leaving the database as it is, I think the thing now is to just go look among any an all backups that are available of the file system, starting with the most recent, until you find one which still has this file present in the /files directory. And then diff that backup's /files folder against the current live site, and just bring over any files in the backup that aren't still on the site. (Not overwriting anything, nor deleting anything; just bringing back "missing" files.) Yes, you might end up bringing over some files which had been intentionally deleted since then, but that's the "lesser evil" at this point.

canonknipser mentioned using a script to "extract all attachment files." That's not really "what you actually want to do", but that script does happen to report "copy failed:" in cases where the physical filename wasn't available in the /files directory. And that part is what you want to know: are there any other attachments still present in the attachments table in the database which no longer physically exist in the /files folder.

Again, you already know that's true right now, and it's not necessary to run that script yet. But after you've found and restored from a backup where attachment ID 4095 was still present in the /files directory, then the question you must answer is "was that all of them, or are there still other files missing I must try and find a backup for?" And this script which enumerates through the attachment table and tries to copy the physical filename will help you identify whether there are still other missing physical files.

BeBop
Registered User
Posts: 45
Joined: Tue Sep 05, 2017 1:07 pm

Re: Done 3.2.2 to 3.2.7 upgrade but inline attachments missing

Post by BeBop » Mon Aug 19, 2019 8:50 pm

DEAR EA117,

Many thanks for all your help above and apologies for the long delay in getting back to you, but Robbell our systems techie has been off sick for a chunk of the month and I am trying as a user to decipher what we need to do next.

Are we looking for a back up taken before we did the upgrade to 3.2.7 when we appear to have lost the attachments and that being so, then can I CONFIRM that we need a back up that contains an intact /files folder named CHMOD 777? Assuming that is what we need, then there must be a specific file within the /files folder itself that contains all the attachment images YES?, that being true, then what is the name of that specific file please?

Speaking as an elderly tech novice, are these back ups likely to have been located within the C Panel itself or on our main lap top that interracts with the host provider? Our host provider only maintains back ups up to 6 days old internally as part of the service, which is distinctly unhelpful This problem was obviously reported a lot earlier than six days ago, but we need to know what we are looking for and the simpleton's guide as to how to find it please.

Thank you for your help in advance and to date.

Kind Regards BeBop

User avatar
EA117
Registered User
Posts: 1037
Joined: Wed Aug 15, 2018 3:23 am
Contact:

Re: Done 3.2.2 to 3.2.7 upgrade but inline attachments missing

Post by EA117 » Mon Aug 19, 2019 10:41 pm

Hello BeBop,

You're probably going to want to either wait for Robbell or hire someone who can do Robbell's work in his absence. Because we're not talking about a process that is commonplace or "simple, if we could just understand two or three steps." It's also potentially true that some of the backups in question are things only Robbell had, or that maybe only Robbell knew had been created.

Note "chmod 777" isn't a file that exists in the /files/ folder. It's a Linux file system permissions command, which was important to the discussion in case the permissions on the /files/ folder or the files currently in the /files/ folder were what was preventing the files from being accessible to phpBB.

We're not talking about just a single file within the /files/ folder backup. Robbell was looking in the phpbb_attachments database table, at the individual attachment IDs, and finding what the "physical filename" was for some of the problematic attachments. Each attachment has it's own unique physical filename that would exist in the /files/ folder. What Robbell found was that the current "broken" attachments simply have their physical files missing from the /files/ folder.

And we're not talking about any specific kind of backup, or time frame for the backup. Just "whatever backups you actually have available." You would want to look at all of them, starting with the most recent, and evaluating "do they contain these missing files?" The backups might have been backups performed through CPanel, and/or we may also be talking about backups that were made simply by FTP'ing the contents of the site.

The proverbial "first step", now that Robbell has identified the specific reason why at least two of the problematic attachments were broken, is to definitively query the database and determine all of the attachments that are broken in this manner. Meaning, query every single attachment ID in the phpbb_attachments database table, examine every single "physical filename" value, and then test whether that specific physical filename actually exists in the /files/ directory.

That is where the use of the script canonknipser mentioned might come in handy. Since even though you're not literally trying to "extract all attachment files" as the script was designed to do, the script will identify when "the physical file needed for this attachment does not exist in the /files/ directory." By capturing the output of that script, you would be able to then use that output to build a list of all the physical file names that are missing. That list of missing physical files in the /files/ folder is what you need in order to then go look in the backups to find "which backup(s) can help me restore these missing files."

Note it's highly likely that Robbell, you, or whomever does the work would want to do this against a local XAMPP-based copy of the site, instead of the risk and slower performance of trying to query or manipulate the database or the backups on the live site. Once the problem has been solved within the local XAMPP-based copy of the site, you'll be able to FTP just the set of missing files up to the /files/ folder of the live production site.

BeBop
Registered User
Posts: 45
Joined: Tue Sep 05, 2017 1:07 pm

Re: Done 3.2.2 to 3.2.7 upgrade but inline attachments missing

Post by BeBop » Sun Aug 25, 2019 7:31 pm

DEAR EA117,

Many thanks for all your help above and apologies for another delay in our response. However, we followed your advice and waited to see if Robbell our systems techie was fit enough to get back on the case. Unfortunately not yet….. So as a user I would like to ask you some questions regarding the procedure you have described.

Does the phpBB board software physically assigns its own internal identifiers to these inline image attachments in order to compare entries in the current phpbb_attachments database table with the content records of the /files folder presently in use? To BE ABLE TO confirm that they do or do not have corresponding images in this edition of the /files folder? The reason behind this question is the fact that the original attachment file names used are not substantially different from one another.

As this comparison is a massive job [POSSIBLY MORE THAN 1,000 inline attachment images DISCONNECTED from posts !] you have recommended the use of a script called canonknipser. From what I can deduce, I think we actually need a utility named "viewexif extension" within the canonknipser utility family? https://www.canonknipser.com/viewtopic.php?f=14&t=1 iin order to establish whether any of the attachment images on the phpbb_attachments database table currently exist in the /file folder. Is this correct and how do we download it?

Continued thanks and Kind Regards
BeBop

User avatar
canonknipser
Registered User
Posts: 2096
Joined: Thu Sep 08, 2011 4:16 am
Location: Germany
Name: Frank Jakobs
Contact:

Re: Done 3.2.2 to 3.2.7 upgrade but inline attachments missing

Post by canonknipser » Sun Aug 25, 2019 8:24 pm

BeBop wrote:
Sun Aug 25, 2019 7:31 pm
you have recommended the use of a script called canonknipser.
The script is not called "canonknipser" - that is my nickname :mrgreen:
The primary purpose of the script rename.php I linked and EA117 referred to, is to copy the files from the attachments folder (this folder is called files on a regular installation) to another folder. Because phpBB uses "cryptic" files names without any file extension, the script copies the files under their original names (the file names the users had on their personal computers when uploading them to a phpBB board).
BeBop wrote:
Sun Aug 25, 2019 7:31 pm
The reason behind this question is the fact that the original attachment file names used are not substantially different from one another.
Yes, that is how phpBB organises attachments. They get a unique, but randomized name, which is 32 characters long for every attachment. The file names are build from the characters 0 to 9 and a to f (which are the "numbers" of the hexadecimal systems), prepended by the numeric value of the user_id from the user who upload the attachment. This filename is stored in the database table attachments. And yes, those file names look very similar to each other ;)

The goal for your site is to find out which of the entries in you table have no physical file present in the files folder. My script can help you to find out (as already mentioned)
EA117 wrote:
Tue Jul 30, 2019 4:25 pm
And that part is what you want to know: are there any other attachments still present in the attachments table in the database which no longer physically exist in the /files folder.
For each file which is copied from your files folder to the destination folder there is a entry in a list if the copy action was successful (Your file is present) or unsuccessful (the file is missing). Just copy the file names of the missing files into a list and use that list to copy the files back from a backup to the files folder.

BeBop wrote:
Sun Aug 25, 2019 7:31 pm
From what I can deduce, I think we actually need a utility named "viewexif extension"
You don't need any other of my extensions, the script is working without any extensions to phpBB.
robbell wrote:
Tue Jul 30, 2019 11:32 am
I have noticed there's an in_message field, everything is set to 0 in that field - could that be indicating a problem?
Don't worry about that - in_message indicates, if a attachment was used in a regular post (value 0) or in a personal message (value 1)

BeBop wrote:
Sun Aug 25, 2019 7:31 pm
As this comparison is a massive job [POSSIBLY MORE THAN 1,000 inline attachment images DISCONNECTED from posts !]
So, you did count the files in the files folder and the entries in the attachment table?
  • If you have a entry in your attachment table, but no likely named file present on your files folder, the script linked above will give you a detailed list of those missing files and you are able to restore them (hopefully) from a backup.
  • if you have a file present in the folder, but not in the attachment table, this can have two causes (apart from the ever present file index.htm and .htaccess:
    1. The file is a so called thumbnail to a attached image file. You can identify the file by the prefix thumb, the rest of the file name is the name of the corresponding image file - don't bother about this "extra" files
    2. The file is a extra file not linked to any entry in the attachment table - don't bother either unless you are running out of disk space
  • If you see orphaned attachments in you acp - this is another point you don't need to bother about. Those orphaned attachment are attachments which where uploaded to a post a user was creating, but never submitted.
Greetings, Frank
phpbb.de support team member
English is not my native language - no support via PM or mail
New arrival - Extensions and scripts for phpBB

Post Reply

Return to “[3.2.x] Support Forum”