[CDB] AWS S3

A place for Extension Authors to post and receive feedback on Extensions still in development. No Extensions within this forum should be used within a live environment!
Anti-Spam Guide
Forum rules
READ: phpBB.com Board-Wide Rules and Regulations

IMPORTANT: Extensions Development rules

IMPORTANT FOR NEEDED EVENTS!!!
If you need an event for your extension please read this for the steps to follow to request the event(s)
User avatar
austin881
Registered User
Posts: 284
Joined: Wed Jan 30, 2008 9:58 pm
Location: Boise, Idaho, USA
Name: Austin Maddox
Contact:

[CDB] AWS S3

Post by austin881 » Fri Sep 02, 2016 4:52 am

Extension Name: AWS S3

Author: AustinMaddox

Extension Description: A phpBB extension that utilizes Amazon Web Services (AWS) Simple Storage Service (S3) as the filesystem for uploaded files. This extension allows you to run multiple servers behind a load balancer without worrying about which servers filesystem the image got uploaded to. No more trying to synchronize the filesystems since any server that handles the request will reference the image from your S3 bucket, not the local filesystem.

Local filesystems are dead.
Remote filesystems are the future.
Cloud is the new normal.
Death to the monolith!

I feel like it is critical in this distributed computing day-in-age to store user-generated content in some sort of easily-accessible cloud storage. Amazon S3 provides secure, durable, highly-scalable cloud storage at a very low cost.

In a perfect world, phpBB would share a philosophy of a filesystem like Laravel does. Laravel provides a powerful filesystem abstraction thanks to the Flysystem PHP package by Frank de Jonge. I understand that splitting uploaded files storage into separate class(es) has been proposed in PHPBB3-14169, but as of the time of this writing it has not yet been merged.

This may only be one step towards a fully distributed phpBB application that is capable of running in a clustered environment; but, at least it is moving in the right direction. By leveraging the AWS cloud we can run phpBB with increased flexibility, scalability and reliability.

Extension Version: 1.0.0

Requirements:
  • This extension requires PHP >= 5.5.0.
  • To successfully send requests to AWS APIs, you need to have a valid set of security credentials called access keys. To learn more about AWS access keys, visit the About Access Keys page in the developer documentation.
  • You must run composer install in the directory of this extension to install the required dependencies.
Features:
  • Utilizes AWS PHP SDK Version 3.
  • Access keys, AWS region, and S3 bucket are configurable via the ACP.
  • When enabled, uploaded attachments will be written to the local filesystem of the server as well as uploaded to the S3 bucket of your choosing.
  • Attachment images on topic pages will reference your images on S3 rather than the local filesystem. (ie. http://your-s3-bucket.s3.amazonaws.com/thumb_2_424f080177ab2cc3b9880b592527d115
  • Attachments that are deleted, will also be deleted from your S3 bucket.
  • Both fullsize and resized thumb images are uploaded to S3.
  • Images are uploaded with public-read ACL so your images are publicly visible via the web.
Future considerations:
  • Possible CloudFront global content delivery network (CDN) distribution configuration.
Related topics: Screenshots:
Image

Extension Download:
https://github.com/AustinMaddox/phpbb-extension-s3/archive/master.zip

I welcome any suggestions, feedback, questions, co-contributors, and pull requests. ;)
https://github.com/AustinMaddox/phpbb-extension-s3
Last edited by austin881 on Mon Jun 12, 2017 1:51 am, edited 8 times in total.
Available for paid phpBB help! PM me.

My Extensions/MODS: 475 Narius Categorized Smilies for phpBB3, Simplified & Compacted All-Members page, Flash Animated Cumulus Tag Cloud, "Hello" Name Tag of Newest User, AWS S3
phpBB portfolio: chevy truck forum, chevy astro van forum

BF79
Registered User
Posts: 272
Joined: Fri Feb 13, 2015 1:28 pm

Re: [ALPHA] AWS S3

Post by BF79 » Fri Sep 02, 2016 9:34 pm

thx and good luck for next!!

Mcclures
Registered User
Posts: 33
Joined: Sun Apr 10, 2016 6:04 am

Re: [ALPHA] AWS S3

Post by Mcclures » Tue Sep 06, 2016 1:08 am

Nice. I use AWS myself. Any Possibility For EC2 Storage?

User avatar
austin881
Registered User
Posts: 284
Joined: Wed Jan 30, 2008 9:58 pm
Location: Boise, Idaho, USA
Name: Austin Maddox
Contact:

Re: [BETA] AWS S3

Post by austin881 » Wed Sep 07, 2016 5:12 am

Thanks BF79.

Mcclures, I'm not sure what you mean. Storing files on EC2 is the default behavior. If you were to host a phpBB site on an EC2 instance on AWS, by default phpBB uses the local filesystem which would be a mounted EBS-optimized EC2 volume and/or instance store storage. Your question doesn't make sense and maybe you could clarify but I will say that this extension was written for the sole purpose of uploading files to S3, not anywhere else.

So has anyone installed and tested this extension? I'm interested to hear your feedback and hope to hear positive reports of people getting it working on their test sites. I've updated the topic to [BETA] status because I believe it is ready to go and if I don't receive show-stopping feedback I'll go ahead and submit it for inclusion in the Extensions Database. :)
Available for paid phpBB help! PM me.

My Extensions/MODS: 475 Narius Categorized Smilies for phpBB3, Simplified & Compacted All-Members page, Flash Animated Cumulus Tag Cloud, "Hello" Name Tag of Newest User, AWS S3
phpBB portfolio: chevy truck forum, chevy astro van forum

User avatar
david63
Registered User
Posts: 16661
Joined: Thu Dec 19, 2002 8:08 am
Location: Lancashire, UK
Name: David Wood
Contact:

Re: [BETA] AWS S3

Post by david63 » Wed Sep 07, 2016 7:09 am

I have not tested this but I do have a question - "What fallback strategies are built in to this extension for any situation when AWS is not accessible?"
David
Remember: You only know what you know and - you don't know what you don't know!
My CDB Contributions | How to install an extension
I will not be accepting translations for any of my extensions in Github - please post any translations in the appropriate topic.
No support requests via PM or email as they will be ignored

User avatar
austin881
Registered User
Posts: 284
Joined: Wed Jan 30, 2008 9:58 pm
Location: Boise, Idaho, USA
Name: Austin Maddox
Contact:

Re: [BETA] AWS S3

Post by austin881 » Sat Sep 17, 2016 5:14 am

S3 has reached a level of maturity such that fallback strategies are not as important as they may have once been. Let me ask you a strange question to help illustrate my point... What fallback strategies are built into phpBB for any situation when the filesystem (ie. Hard Drive) is not accessible?

S3 was originally launched on March 14, 2006; 10 years ago! I've been using S3 a long time, and back in the early days of S3 "fallback strategies" were an important part of the implementation. Although we all know that no Software-as-a-Service (SaaS) is perfect, I don't believe we have reason to be concerned. It is really a personal decision to use this extension and all depends on your level of comfort with a third party service like AWS. As for me, like I said I've been using S3 for years. I'm comfortable.
Amazon S3 gives developers access to the same highly scalable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites! S3 Standard is designed for 99.99% availability and backed by the Amazon S3 Service Level Agreement.
Amazon S3 redundantly stores your objects on multiple devices across multiple facilities in an Amazon S3 Region. The service is designed to sustain concurrent device failures by quickly detecting and repairing any lost redundancy. When processing a request to store data, the service will redundantly store your object across multiple facilities before returning SUCCESS.
I'm not sure if that answered your question or if you were just wondering what happens when an uploaded attachment fails... Does it display an error message or what not. The answer is: It fails just like any file upload would, the AJAX that was fired returns a false response and the user can visually see from the progress bar that the upload did not succeed.
Available for paid phpBB help! PM me.

My Extensions/MODS: 475 Narius Categorized Smilies for phpBB3, Simplified & Compacted All-Members page, Flash Animated Cumulus Tag Cloud, "Hello" Name Tag of Newest User, AWS S3
phpBB portfolio: chevy truck forum, chevy astro van forum

User avatar
k0nsl
Registered User
Posts: 88
Joined: Mon Mar 25, 2013 11:17 pm
Location: Scandinavia
Name: Eduard Drehner
Contact:

Re: [BETA] AWS S3

Post by k0nsl » Sat Sep 17, 2016 8:53 am

Thank you for the work on this extension. I haven't tried it yet but I do have a question ! :mrgreen:
Do you think it would be difficult to add support for alternatives to S3 (i.e S3 compatible storage)? For example, I host my own and it would've been nice to be able to use that rather than paying Amazon in order to use their service :)

Best wishes,
-k0nsl
I'm a programmer, server administrator, designer, spokesman and factotum of a wide-variety of web pages.

User avatar
austin881
Registered User
Posts: 284
Joined: Wed Jan 30, 2008 9:58 pm
Location: Boise, Idaho, USA
Name: Austin Maddox
Contact:

Re: [BETA] AWS S3

Post by austin881 » Mon Sep 19, 2016 9:03 pm

k0nsl, using the same event listeners and concept I've outlined in this extension I don't think it would be too difficult for someone to write a similar extension that uploads files to a different service. Provided, of course, that the other service has RESTful API functionality similar to S3.

Like I have said before, this extension is designed for and has the sole purpose of interacting with Amazon Web Services S3 service. I have no plans to add support for any other services or storage options other than S3.
Available for paid phpBB help! PM me.

My Extensions/MODS: 475 Narius Categorized Smilies for phpBB3, Simplified & Compacted All-Members page, Flash Animated Cumulus Tag Cloud, "Hello" Name Tag of Newest User, AWS S3
phpBB portfolio: chevy truck forum, chevy astro van forum

User avatar
John connor
Registered User
Posts: 2293
Joined: Fri Nov 14, 2014 5:14 pm
Location: U S Of A
Name: Aaron
Contact:

Re: [BETA] AWS S3

Post by John connor » Tue Sep 20, 2016 2:26 am

Thanks! I'll have a look at this.

As to the question if and that's a big IF the S3 doesn't respond, does the local storage take over? I'm thinking not. Also, if I remove S3, do my attachments go back to normal? i.e. Not using S3?

Edit-

Another question. Will this send the current attachments to S3?

Edit 2- Tried to upload the extension and got a The “AustinMaddox/phpbb-extension-s3” extension is not valid.
Meta field name is invalid.

User avatar
John connor
Registered User
Posts: 2293
Joined: Fri Nov 14, 2014 5:14 pm
Location: U S Of A
Name: Aaron
Contact:

Re: [BETA] AWS S3

Post by John connor » Wed Sep 21, 2016 5:49 am

Looks like this meta error could be due to the composer.json Googleing this.

User avatar
austin881
Registered User
Posts: 284
Joined: Wed Jan 30, 2008 9:58 pm
Location: Boise, Idaho, USA
Name: Austin Maddox
Contact:

Re: [BETA] AWS S3

Post by austin881 » Wed Sep 21, 2016 2:03 pm

John connor wrote: As to the question if and that's a big IF the S3 doesn't respond, does the local storage take over? I'm thinking not. Also, if I remove S3, do my attachments go back to normal? i.e. Not using S3?
This extension doesn't actually take over the default behavior of phpBB writing to the local filesystem. When this extension is enabled, it additionally uploads files to S3. This is simply an add-on benefit. The extension just bolts on to the event and takes that uploaded file and sends it to two places... local filesystem and S3.

I thought was pretty clear about that in the first post where I listed the features...
AustinMaddox wrote:When enabled, uploaded attachments will be written to the local filesystem of the server as well as uploaded to the S3 bucket of your choosing.
John connor wrote: Another question. Will this send the current attachments to S3?
No, it will not.
John connor wrote: Tried to upload the extension and got a The “AustinMaddox/phpbb-extension-s3” extension is not valid.
Meta field name is invalid.
:roll: I don't think it likes the capitalization of my GitHub name or something. Actually, it might not like the dashes in the project name. According to Extension Validation Policy the name should not have any dashes or underscores. Maybe I should rename it to AustinMaddox/phpbbextensions3? That is kinda dumb. The EPV fails with the error "Error: Packaging structure doesn't meet the extension DB policies." Perhaps it is a similar issue to this topic? Anyone else know what the problem is there?

I'm looking into this more and will report back when I figure out what the issue is.
Available for paid phpBB help! PM me.

My Extensions/MODS: 475 Narius Categorized Smilies for phpBB3, Simplified & Compacted All-Members page, Flash Animated Cumulus Tag Cloud, "Hello" Name Tag of Newest User, AWS S3
phpBB portfolio: chevy truck forum, chevy astro van forum

User avatar
k0nsl
Registered User
Posts: 88
Joined: Mon Mar 25, 2013 11:17 pm
Location: Scandinavia
Name: Eduard Drehner
Contact:

Re: [BETA] AWS S3

Post by k0nsl » Thu Sep 22, 2016 2:23 am

Thank you for the answer. I haven't looked at your code yet, but as you say, theoretically there shouldn't be a problem to use it for any other S3 compatible solution.
I will check when I get some spare time.

-k0nsl
austin881 wrote:k0nsl, using the same event listeners and concept I've outlined in this extension I don't think it would be too difficult for someone to write a similar extension that uploads files to a different service. Provided, of course, that the other service has RESTful API functionality similar to S3.

Like I have said before, this extension is designed for and has the sole purpose of interacting with Amazon Web Services S3 service. I have no plans to add support for any other services or storage options other than S3.
I'm a programmer, server administrator, designer, spokesman and factotum of a wide-variety of web pages.

User avatar
John connor
Registered User
Posts: 2293
Joined: Fri Nov 14, 2014 5:14 pm
Location: U S Of A
Name: Aaron
Contact:

Re: [BETA] AWS S3

Post by John connor » Thu Sep 22, 2016 3:43 am

austin881 wrote:
John connor wrote: As to the question if and that's a big IF the S3 doesn't respond, does the local storage take over? I'm thinking not. Also, if I remove S3, do my attachments go back to normal? i.e. Not using S3?
This extension doesn't actually take over the default behavior of phpBB writing to the local filesystem. When this extension is enabled, it additionally uploads files to S3. This is simply an add-on benefit. The extension just bolts on to the event and takes that uploaded file and sends it to two places... local filesystem and S3.

I thought was pretty clear about that in the first post where I listed the features...
AustinMaddox wrote:When enabled, uploaded attachments will be written to the local filesystem of the server as well as uploaded to the S3 bucket of your choosing.
John connor wrote: Another question. Will this send the current attachments to S3?
No, it will not.
John connor wrote: Tried to upload the extension and got a The “AustinMaddox/phpbb-extension-s3” extension is not valid.
Meta field name is invalid.
:roll: I don't think it likes the capitalization of my GitHub name or something. Actually, it might not like the dashes in the project name. According to Extension Validation Policy the name should not have any dashes or underscores. Maybe I should rename it to AustinMaddox/phpbbextensions3? That is kinda dumb. The EPV fails with the error "Error: Packaging structure doesn't meet the extension DB policies." Perhaps it is a similar issue to this topic? Anyone else know what the problem is there?

I'm looking into this more and will report back when I figure out what the issue is.
Thank you for your responses. I just created a S3 account all set to try this and stumbled upon that error while trying to upload the extension to my board. I couldn't activate as I got that error. I look forward to a fix. Thanks again.


Edit-

I went and renamed all instances of AustinMaddox/phpbb-extension-s3 to austinmaddox/phpbbextensions3. I used the program Fileseek to accomplish this. When I went to test the extesion in a local install using Xampp and clicking on enable S3 I got this error:

Fatal error: Cannot redeclare class austinmaddox\s3\migrations\release_0_1_0 in C:\xampp\htdocs\new\ext\austinmaddox\phpbbextensions3\migrations\release_0_1_0.php on line 43

Not sure if that's a legit error or not or I didn't do something right renaming the text in all the files that had AustinMaddox listed. Just thought I'd throw this out there. I thought my method would work, but I know nothing about PHP.

User avatar
John connor
Registered User
Posts: 2293
Joined: Fri Nov 14, 2014 5:14 pm
Location: U S Of A
Name: Aaron
Contact:

Re: [BETA] AWS S3

Post by John connor » Fri Sep 23, 2016 10:40 pm

Any news on this fix? I signed up for S3 and would love to use this extension. :ugeek:

User avatar
John connor
Registered User
Posts: 2293
Joined: Fri Nov 14, 2014 5:14 pm
Location: U S Of A
Name: Aaron
Contact:

Re: [BETA] AWS S3

Post by John connor » Wed Sep 28, 2016 6:31 am

I guess getting an Amazon S3 account was a bad idea since this extension doesn't work.

Locked

Return to “Extensions in Development”