[CDB] AWS S3

[austin881]

Extension Name: AWS S3

Author: AustinMaddox

Extension Description: A phpBB extension that utilizes Amazon Web Services (AWS) Simple Storage Service (S3) as the filesystem for uploaded files. This extension allows you to run multiple servers behind a load balancer without worrying about which servers filesystem the image got uploaded to. No more trying to synchronize the filesystems since any server that handles the request will reference the image from your S3 bucket, not the local filesystem.

Local filesystems are dead.
Remote filesystems are the future.
Cloud is the new normal.
Death to the monolith!

I feel like it is critical in this distributed computing day-in-age to store user-generated content in some sort of easily-accessible cloud storage. Amazon S3 provides secure, durable, highly-scalable cloud storage at a very low cost.

In a perfect world, phpBB would share a philosophy of a filesystem like Laravel does. Laravel provides a powerful filesystem abstraction thanks to the Flysystem PHP package by Frank de Jonge. I understand that splitting uploaded files storage into separate class(es) has been proposed in PHPBB3-14169, but as of the time of this writing it has not yet been merged.

This may only be one step towards a fully distributed phpBB application that is capable of running in a clustered environment; but, at least it is moving in the right direction. By leveraging the AWS cloud we can run phpBB with increased flexibility, scalability and reliability.

Extension Version: 1.0.0

Requirements:

• This extension requires PHP >= 5.5.0.
• To successfully send requests to AWS APIs, you need to have a valid set of security credentials called access keys. To learn more about AWS access keys, visit the About Access Keys page in the developer documentation.
• You must run composer install in the directory of this extension to install the required dependencies.

Features:

• Utilizes AWS PHP SDK Version 3.
• Access keys, AWS region, and S3 bucket are configurable via the ACP.
• When enabled, uploaded attachments will be written to the local filesystem of the server as well as uploaded to the S3 bucket of your choosing.
• Attachment images on topic pages will reference your images on S3 rather than the local filesystem. (ie. http://your-s3-bucket.s3.amazonaws.com/thumb_2_424f080177ab2cc3b9880b592527d115
• Attachments that are deleted, will also be deleted from your S3 bucket.
• Both fullsize and resized thumb images are uploaded to S3.
• Images are uploaded with public-read ACL so your images are publicly visible via the web.

Future considerations:

• Possible CloudFront global content delivery network (CDN) distribution configuration.

Related topics:

• [ABD] Using Amazon S3 Simple Storage
• Amazon S3 cloud storage extension
• AWS SDK for PHP 3.x
• AWS Tips I Wish I'd Known Before I Started

Screenshots:


Extension Download:
https://github.com/AustinMaddox/phpbb-extension-s3/archive/master.zip

I welcome any suggestions, feedback, questions, co-contributors, and pull requests.
https://github.com/AustinMaddox/phpbb-extension-s3

[BF79]

thx and good luck for next!!

[Mcclures]

Nice. I use AWS myself. Any Possibility For EC2 Storage?

[austin881]

Thanks BF79.

Mcclures, I'm not sure what you mean. Storing files on EC2 is the default behavior. If you were to host a phpBB site on an EC2 instance on AWS, by default phpBB uses the local filesystem which would be a mounted EBS-optimized EC2 volume and/or instance store storage. Your question doesn't make sense and maybe you could clarify but I will say that this extension was written for the sole purpose of uploading files to S3, not anywhere else.

So has anyone installed and tested this extension? I'm interested to hear your feedback and hope to hear positive reports of people getting it working on their test sites. I've updated the topic to [BETA] status because I believe it is ready to go and if I don't receive show-stopping feedback I'll go ahead and submit it for inclusion in the Extensions Database.

[david63]

I have not tested this but I do have a question - "What fallback strategies are built in to this extension for any situation when AWS is not accessible?"

[austin881]

S3 has reached a level of maturity such that fallback strategies are not as important as they may have once been. Let me ask you a strange question to help illustrate my point... What fallback strategies are built into phpBB for any situation when the filesystem (ie. Hard Drive) is not accessible?

S3 was originally launched on March 14, 2006; 10 years ago! I've been using S3 a long time, and back in the early days of S3 "fallback strategies" were an important part of the implementation. Although we all know that no Software-as-a-Service (SaaS) is perfect, I don't believe we have reason to be concerned. It is really a personal decision to use this extension and all depends on your level of comfort with a third party service like AWS. As for me, like I said I've been using S3 for years. I'm comfortable.

Amazon S3 gives developers access to the same highly scalable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites! S3 Standard is designed for 99.99% availability and backed by the Amazon S3 Service Level Agreement.

Amazon S3 redundantly stores your objects on multiple devices across multiple facilities in an Amazon S3 Region. The service is designed to sustain concurrent device failures by quickly detecting and repairing any lost redundancy. When processing a request to store data, the service will redundantly store your object across multiple facilities before returning SUCCESS.

I'm not sure if that answered your question or if you were just wondering what happens when an uploaded attachment fails... Does it display an error message or what not. The answer is: It fails just like any file upload would, the AJAX that was fired returns a false response and the user can visually see from the progress bar that the upload did not succeed.

[k0nsl]

Thank you for the work on this extension. I haven't tried it yet but I do have a question !
Do you think it would be difficult to add support for alternatives to S3 (i.e S3 compatible storage)? For example, I host my own and it would've been nice to be able to use that rather than paying Amazon in order to use their service

Best wishes,
-k0nsl

[austin881]

k0nsl, using the same event listeners and concept I've outlined in this extension I don't think it would be too difficult for someone to write a similar extension that uploads files to a different service. Provided, of course, that the other service has RESTful API functionality similar to S3.

Like I have said before, this extension is designed for and has the sole purpose of interacting with Amazon Web Services S3 service. I have no plans to add support for any other services or storage options other than S3.

[2600]

Thanks! I'll have a look at this.

As to the question if and that's a big IF the S3 doesn't respond, does the local storage take over? I'm thinking not. Also, if I remove S3, do my attachments go back to normal? i.e. Not using S3?

Edit-

Another question. Will this send the current attachments to S3?

Edit 2- Tried to upload the extension and got a The “AustinMaddox/phpbb-extension-s3” extension is not valid.
Meta field name is invalid.

[2600]

Looks like this meta error could be due to the composer.json Googleing this.

[austin881]

As to the question if and that's a big IF the S3 doesn't respond, does the local storage take over? I'm thinking not. Also, if I remove S3, do my attachments go back to normal? i.e. Not using S3?

This extension doesn't actually take over the default behavior of phpBB writing to the local filesystem. When this extension is enabled, it additionally uploads files to S3. This is simply an add-on benefit. The extension just bolts on to the event and takes that uploaded file and sends it to two places... local filesystem and S3.

I thought was pretty clear about that in the first post where I listed the features...

When enabled, uploaded attachments will be written to the local filesystem of the server as well as uploaded to the S3 bucket of your choosing.

Another question. Will this send the current attachments to S3?

No, it will not.

Tried to upload the extension and got a The “AustinMaddox/phpbb-extension-s3” extension is not valid.
Meta field name is invalid.

I don't think it likes the capitalization of my GitHub name or something. Actually, it might not like the dashes in the project name. According to Extension Validation Policy the name should not have any dashes or underscores. Maybe I should rename it to AustinMaddox/phpbbextensions3? That is kinda dumb. The EPV fails with the error "Error: Packaging structure doesn't meet the extension DB policies." Perhaps it is a similar issue to this topic? Anyone else know what the problem is there?

I'm looking into this more and will report back when I figure out what the issue is.

[k0nsl]

Thank you for the answer. I haven't looked at your code yet, but as you say, theoretically there shouldn't be a problem to use it for any other S3 compatible solution.
I will check when I get some spare time.

-k0nsl

k0nsl, using the same event listeners and concept I've outlined in this extension I don't think it would be too difficult for someone to write a similar extension that uploads files to a different service. Provided, of course, that the other service has RESTful API functionality similar to S3.

Like I have said before, this extension is designed for and has the sole purpose of interacting with Amazon Web Services S3 service. I have no plans to add support for any other services or storage options other than S3.

[2600]

As to the question if and that's a big IF the S3 doesn't respond, does the local storage take over? I'm thinking not. Also, if I remove S3, do my attachments go back to normal? i.e. Not using S3?

This extension doesn't actually take over the default behavior of phpBB writing to the local filesystem. When this extension is enabled, it additionally uploads files to S3. This is simply an add-on benefit. The extension just bolts on to the event and takes that uploaded file and sends it to two places... local filesystem and S3.

I thought was pretty clear about that in the first post where I listed the features...

When enabled, uploaded attachments will be written to the local filesystem of the server as well as uploaded to the S3 bucket of your choosing.

Another question. Will this send the current attachments to S3?

No, it will not.

Tried to upload the extension and got a The “AustinMaddox/phpbb-extension-s3” extension is not valid.
Meta field name is invalid.

I don't think it likes the capitalization of my GitHub name or something. Actually, it might not like the dashes in the project name. According to Extension Validation Policy the name should not have any dashes or underscores. Maybe I should rename it to AustinMaddox/phpbbextensions3? That is kinda dumb. The EPV fails with the error "Error: Packaging structure doesn't meet the extension DB policies." Perhaps it is a similar issue to this topic? Anyone else know what the problem is there?

I'm looking into this more and will report back when I figure out what the issue is.

Thank you for your responses. I just created a S3 account all set to try this and stumbled upon that error while trying to upload the extension to my board. I couldn't activate as I got that error. I look forward to a fix. Thanks again.


Edit-

I went and renamed all instances of AustinMaddox/phpbb-extension-s3 to austinmaddox/phpbbextensions3. I used the program Fileseek to accomplish this. When I went to test the extesion in a local install using Xampp and clicking on enable S3 I got this error:

Fatal error: Cannot redeclare class austinmaddox\s3\migrations\release_0_1_0 in C:\xampp\htdocs\new\ext\austinmaddox\phpbbextensions3\migrations\release_0_1_0.php on line 43

Not sure if that's a legit error or not or I didn't do something right renaming the text in all the files that had AustinMaddox listed. Just thought I'd throw this out there. I thought my method would work, but I know nothing about PHP.

[2600]

Any news on this fix? I signed up for S3 and would love to use this extension.

[2600]

I guess getting an Amazon S3 account was a bad idea since this extension doesn't work.