10K File limit for FTP

Discussion of non-phpBB related topics with other phpBB.com users.
Forum rules
General Discussion is a bonus forum for discussion of non-phpBB related topics with other phpBB.com users. All site rules apply.
Post Reply
User avatar
thecoalman
Community Team Member
Community Team Member
Posts: 5850
Joined: Wed Dec 22, 2004 3:52 am
Location: Pennsylvania, U.S.A.
Contact:

10K File limit for FTP

Post by thecoalman »

Don't know how common this issue is but I just became aware of something that could really cause a disaster when backing up phpBB's files directory if there is more than ten thousands files. I don't know if this is something new and I don't recall it being issue in the past with regular FTP. I've been using SFTP for ages now and it doesn't affect that.

Host Godaddy
FTP Client Winscp
Connection: Regular old FTP

It will only list 10K files and even if you select the folder itself to download it will only contain 10K files, at least that's the results I got with WinSCP. There is no warning in the FTP client unless I somehow missed it and the only indication anything is up is the FTP logs.
226 Output truncated to 10000 matches
Based on what I'm reading that file count can be more or less but that's the default. The obvious solution is use SFTP but if you are using FTP this is a heads up.
“Results! Why, man, I have gotten a lot of results! I have found several thousand things that won’t work.”

Attributed - Thomas Edison
User avatar
david63
Registered User
Posts: 20646
Joined: Thu Dec 19, 2002 8:08 am

Re: 10K File limit for FTP

Post by david63 »

If I am not mistaken this is a server file configuration/limitation issue that has been around for years, and as far as I am aware with phpBB is really only an issue with the files folder - which is one of the reasons why there is a change pending to be able to use multiple folders for attachments as the default configuration on many servers is 10k.

There are plenty of articles about this issue with a Google search
User avatar
thecoalman
Community Team Member
Community Team Member
Posts: 5850
Joined: Wed Dec 22, 2004 3:52 am
Location: Pennsylvania, U.S.A.
Contact:

Re: 10K File limit for FTP

Post by thecoalman »

Yes it's server configuration and I should of mentioned that. I have been using SFTP for many years now and it's something that must have been implemented in that time. I know for fact I would of been doing operations on folders with more than 10K files when I was still using regular FTP.

Point is that there is no warning for this, at least any I saw while using WinSCP. It's something you can easily overlook especially if you select a folder for download. Minimally the FTP client should be warning you about this when it receives that message.
“Results! Why, man, I have gotten a lot of results! I have found several thousand things that won’t work.”

Attributed - Thomas Edison
User avatar
AmigoJack
Registered User
Posts: 6106
Joined: Tue Jun 15, 2010 11:33 am
Location: グリーン ヒル ゾーン
Contact:

Re: 10K File limit for FTP

Post by AmigoJack »

thecoalman wrote: Mon Feb 07, 2022 1:55 pmthe FTP client should be warning you about this when it receives that message.
It can't, since there is no standard for this case. Even the code 226 is questionable, as your request both failed and succeeded - 214 is more appropriate. FTP return codes don't even have distinctive meanings like HTTP has - you still have to expect that every server uses his own text or what he thinks is the correct code for it.
User avatar
thecoalman
Community Team Member
Community Team Member
Posts: 5850
Joined: Wed Dec 22, 2004 3:52 am
Location: Pennsylvania, U.S.A.
Contact:

Re: 10K File limit for FTP

Post by thecoalman »

If the code is not a standard then it needs to be or some kind of alert. If you select a folder to download surely the expectation is you will get the entire contents in that folder. This is nearly as bad as the Filezilla issue. The worst thing about this is people using regular FTP are going to be the least likely to know about it. I didn't know about this but I probably haven't used FTP in a decade.

The only reason I'm aware of it is because I've spent the past few days trying to help someone recover multiple missing files. I knew something was up because there was no files prefixed with thumb_. The owner of this forum was diligently making "backups" only to find out they were missing almost 8000 source files. In their attempt to fix some other errors they deleted all of their board files and restored a "backup", *poof* goes 8000 files.
“Results! Why, man, I have gotten a lot of results! I have found several thousand things that won’t work.”

Attributed - Thomas Edison
User avatar
AmigoJack
Registered User
Posts: 6106
Joined: Tue Jun 15, 2010 11:33 am
Location: グリーン ヒル ゾーン
Contact:

Re: 10K File limit for FTP

Post by AmigoJack »

thecoalman wrote: Mon Feb 14, 2022 10:58 pmit needs to be or some kind of alert
Well, you got that kind of alert already - it's just not that you want to see it as "alert".

I know listing limits are an issue, but what you want is next to magic. You could say "I need an FTP client where I can customize what should happen when the server's response text contains this or that" and then you start thinking about match words like "truncate" or "limit" and actions like "show me a dialog window". Until you meet a server that doesn't even talk English...

To be the least help: round folder item counts like 1000, 2000 or 5000 just indicate a limit and should be investigated.
Post Reply

Return to “General Discussion”