[ALPHA TESTING] Mailing list replication feature

A place for MOD Authors to post and receive feedback on MODs still in development. No MODs within this forum should be used within a live environment! No new topics are allowed in this forum.
Forum rules
READ: phpBB.com Board-Wide Rules and Regulations

IMPORTANT: MOD Development Forum rules

On February 1, 2009 this forum will be set to read only as part of retiring of phpBB2.
Locked
danyblue
Registered User
Posts: 337
Joined: Fri Apr 12, 2002 4:48 pm
Contact:

[ALPHA TESTING] Mailing list replication feature

Post by danyblue » Fri Apr 12, 2002 4:53 pm

Hi,
i would like to know it it would be possible to developp a mode that could log into a pop3 mail account and retrieve all mails there.
Based on the principle that the existing mails are from a mail list introduce them into the database as threads. All replies to the first mail would then be reply to the threads.
This could be a nice feature to make the transition between the maillist world and the forum one.

Regards,
Danblue
Last edited by danyblue on Thu Aug 29, 2002 5:53 pm, edited 2 times in total.

drewe2000
Registered User
Posts: 19
Joined: Sun Apr 07, 2002 4:19 am
Contact:

I would be interested in this as well...

Post by drewe2000 » Fri Apr 12, 2002 9:45 pm

Even simpler than coding up a system to log into an email account and download emails would be coding some scripts to parse a standard mbox format file. (although having it check an actual email account for new messags would be sweet)

It could then store each individual email in the database as a separate post to the forum, and then connect the threads together.

This should be relatively straightforward, and would be nice. It would not replace the forums at my site - I wouldn't even allow visitors to post. It would simply serve as a very nice GUI to the mailing list archive.

I *may* be able to get something like this working, but not until sometime this summer when classes are finished. :-)

Anyone else have any thoughts? Want to set up a group effort on this?
--Andrew

http://macdigitalaudio.com
news and information for your ears

tobiaseigen
Registered User
Posts: 37
Joined: Sun Apr 14, 2002 2:51 pm
Location: Takoma Park, MD USA
Contact:

mailing list archiving using phpbb

Post by tobiaseigen » Sun Apr 14, 2002 7:02 pm

Yeah, I'd be very interested in joining a group to discuss this type of mod.

I'm interested in establishing a yahoogroups type service for African non-profits - right now I have a mailman mailing list server which is set up to work with mhonarc to archive messages. The problem with mhonarc is that the archives it produces are not dynamic enough for online discussions and that it is a major hassle to manage the online archives. Of course it's also not searchable. :-)

Mailman produces MBOX archives for all lists by default, so I have loads of mbox files we could work with, going back over a year. Mailman even has a facility to directly call a script when a message has been received for archiving, which I have working very nicely with mhonarc. I would be psyched to get my hands on a script for posting incoming messages to phpbb boards. I have seen similar things for mojomail and phorum, but none come close to what phpbb offers in an online interface.

Qmail also enables other ways to handle incoming mail via dedicated subdomains (ie. listname@lists.kabissa.org), which I am using very successfully for mailman.

I've seen various arguments against "two-way emailing" between phpbb/mailing lists, but I still think it's an important mod to make. Mailman does a great job of managing the email distribution and blocking unwanted postings, and phpbb is a perfect online interface. I'd be glad if we could get more people interested.

All of this is particularly relevant for Kabissa because we serve people who work in parts of the world where access to the internet is difficult - people really have to be able to participate by email if they want to since it's sometimes impossible to get reliable access to the web. I'd like to eventually offer the web archives on cdrom too for offline searching/reading.

Check out http://www.kabissa.org to see how I'm using mhonarc already, and please get in touch with me if you're interested in this kind of mod.

Cheers,

Tobias
Kabissa - Space for change in Africa
http://www.kabissa.org

danyblue
Registered User
Posts: 337
Joined: Fri Apr 12, 2002 4:48 pm
Contact:

Maillist Manager Mod

Post by danyblue » Sun Apr 14, 2002 9:36 pm

I have tried to setup this kind of feature using monharc and other tools liked it, but I have not been able to use them accordingly to what I have in Mind.
My latest try is using a program called Popper which can retrieve mails from a mailbox and save them into a mysql database. Then I am trying to implement a function that try to classify the mails based on simple rule.
If "subject" begins by "re:" them it must represent a reply to a previous mail and there so it should be already a thread in a forum. If not, it will ask the admin to define a category and forum to post the message.
I am not a very expert programmer in Php and in the PHPBB code. So help from people which would see a relevance in this would be fine.

Regards,
Danyblue

tobiaseigen
Registered User
Posts: 37
Joined: Sun Apr 14, 2002 2:51 pm
Location: Takoma Park, MD USA
Contact:

Re: Maillist Manager Mod

Post by tobiaseigen » Sun Apr 14, 2002 9:51 pm

danyblue wrote: I have tried to setup this kind of feature using monharc and other tools liked it, but I have not been able to use them accordingly to what I have in Mind.
My latest try is using a program called Popper which can retrieve mails from a mailbox and save them into a mysql database. Then I am trying to implement a function that try to classify the mails based on simple rule.
If "subject" begins by "re:" them it must represent a reply to a previous mail and there so it should be already a thread in a forum. If not, it will ask the admin to define a category and forum to post the message.
I am not a very expert programmer in Php and in the PHPBB code. So help from people which would see a relevance in this would be fine.

I'm not sure if what you're describing is a useful way to approach the problem - unless this popper program is particularly good. I think it would be better to handle the individual messages as they are handled by mailman through a script that feeds the contents of the message into a phpbb forum.

What I like about mhonarc is that it can handle both an historical record in an MBOX file and individual messages as they are piped through.

The other part of the equation doesn't seem to be such a big deal - as I recall it is quite easy to modify phpbb to mail out the contents of a posting to an email address associated with each forum.

Cheers,

Tobias
Kabissa - Space for change in Africa
http://www.kabissa.org

danyblue
Registered User
Posts: 337
Joined: Fri Apr 12, 2002 4:48 pm
Contact:

Mailist

Post by danyblue » Mon Apr 15, 2002 10:31 am

Hi,
Do you have any code that can be shared as to be able to see your possible solution'

regards
Danyblue

tobiaseigen
Registered User
Posts: 37
Joined: Sun Apr 14, 2002 2:51 pm
Location: Takoma Park, MD USA
Contact:

Re: Mailist

Post by tobiaseigen » Tue Apr 16, 2002 9:24 am

danyblue wrote: Hi,
Do you have any code that can be shared as to be able to see your possible solution'


Are you referring to my post? I experimented with sending out phpbb posts to an email address some time ago, but not with phpbb2. It was quite easy as I recall - you just add a mail() function to the appropriate part of the posting script. There should be something in these forums on that.

I had set up the phpbb forums with names equivalent to my list names, ie. 'pambazuka-news' for the pambazuka-news mailing list. I have a mailman script that then handles all mail to pambazuka-news@lists.kabissa.org. I hacked the posting script to build a message from information in the database - ie. the To address was [forumname]@lists.kabissa.org and the From address is the email address of the poster in phpbb, which would have to be in the subscriber list to post (depending on how it's configured in mailman - this could be configured in both mailman and phpbb depending on what the intended purpose of the list is). The subject line and message body could also be specified using the mail() function.

The way I had it configured, phpbb just sent out the latest posting to the list - subscribers could then click on the link to see the full topic thread, or request it via www4mail@kabissa.org to be delivered to your mailbox. The part that is missing is posting mailman messages to phpbb forums. This is crucial, but it should be possible to hack something out in a perl script that parses the message and dumps the contents into the phpbb database.

It would be useful if attachments could also be handled properly in both directions, ie. by building MIME messages.

Hope this helps - let me know how you get on. I'd be really glad if a phpbb programmer could get involved in creating this mod.

Cheers,

Tobias
Kabissa - Space for change in Africa
http://www.kabissa.org

danyblue
Registered User
Posts: 337
Joined: Fri Apr 12, 2002 4:48 pm
Contact:

maillist

Post by danyblue » Tue Apr 16, 2002 3:45 pm

Hi,
can you share the code you have developped? it would be interresting to have a reference to see how you handle the insert in the Phpbb database, it is where i am stuck.


Regards,

Danyblue

danyblue
Registered User
Posts: 337
Joined: Fri Apr 12, 2002 4:48 pm
Contact:

form a group

Post by danyblue » Wed Apr 17, 2002 11:15 am

Does any one want to make a group to work over this feature?
We could specify the tools to be used, how to use it, etc...
We would probably need the assistance of some with knowledges of PHPBB template and theme integration.

Who is in?

regards,
Danyblue

tobiaseigen
Registered User
Posts: 37
Joined: Sun Apr 14, 2002 2:51 pm
Location: Takoma Park, MD USA
Contact:

group for developing maillist <-> phpbb mod

Post by tobiaseigen » Wed Apr 17, 2002 2:16 pm

Hi Danyblue - I'd be glad to join such a group. Shall we open a topic for this purpose here - can you take the lead? I would be glad to provide expertise in qmail and mailman, and have plenty of thoughts on how this might work. We can use the kabissa.org installation (a fresh phpbb2.0.0 install) for experimentation.

Cheers,

Tobias
Kabissa - Space for change in Africa
http://www.kabissa.org

tobiaseigen
Registered User
Posts: 37
Joined: Sun Apr 14, 2002 2:51 pm
Location: Takoma Park, MD USA
Contact:

Re: maillist

Post by tobiaseigen » Wed Apr 17, 2002 2:26 pm

danyblue wrote: can you share the code you have developped? it would be interresting to have a reference to see how you handle the insert in the Phpbb database, it is where i am stuck.

This is not something I have figured out, although it has been figured out by others - ie. phorum. The easiest thing may be to write a script which parses the message. Mailman has a function to run an "external archiver" when a message has been approved and distributed. On my system this is mhonarc as follows:

PUBLIC_EXTERNAL_ARCHIVER = '/usr/local/bin/mhonarc -add -outdir /var/mhonarc/archives/%(listname)s -rcfile mhon
arc.rc -htmlext php -idxfname index.php -title %(listname)s -reverse >> /tmp/out_external_archiver'

The %(listname)s above is replaced with the listname. It works quite well, you can see the results at http://www.kabissa.org. The problem with mhonarc is that it is difficult to manage the messages (ie. to delete them or to edit the text), to manage permissions, and it's a real pain to read a thread or to search it.

I'd like to be able to call a script to post messages to phpbb forums in the same way that I can call mhonarc. Mhonarc also has a mode where it can read in the contents of an MBOX file - it is essential for me to also be able to do this because of the huge volume of archives I now have for lists hosted on Kabissa.

If a phpbb programmer would be interested in working on this project with us, that would be really great!

Cheers,

Tobias

Cheers,

Tobias
Kabissa - Space for change in Africa
http://www.kabissa.org

tobiaseigen
Registered User
Posts: 37
Joined: Sun Apr 14, 2002 2:51 pm
Location: Takoma Park, MD USA
Contact:

mail2mysql perl script

Post by tobiaseigen » Thu Apr 18, 2002 7:46 am

Hi all,

For those ambitious folks interested in this mod, there's a perl script I found which works with mhonarc to take an mbox file and post the messages in it to a mysql database:

Here's the link:

http://www.blazonry.com/perl/mail2mysql.php

It seems that it should be possible to adapt this quite easily to post incoming messages to phpbb forums.

Cheers,

Tobias
Kabissa - Space for change in Africa
http://www.kabissa.org

danyblue
Registered User
Posts: 337
Joined: Fri Apr 12, 2002 4:48 pm
Contact:

Basic funcionalities

Post by danyblue » Thu Apr 18, 2002 8:17 am

Hi tobiaseigen,
in my case i think that the following functionalities should be included in the mod.

1 - take mails from the mbox file and gets them into mysql.
2 - Parse the resulting messages against the Phpbb database and look matching subjects.
3 - Look at sender and search in the user database for this user (probably based on the name of the sender or on the e-mail), if not found automatically register the user.
4 - If the message is not found on the database, display the message in the screen with a dropdown box containing the forums available, that you can indicate manually where to include the messages that are not recognizer. I am basic this point on the fact that for me a mail list is just like a category where several discussions are taking place and that i want to separte through several forum in order to classify them.
5 - If a new message is posted directly in the forum, sends it automatically to the corresponding maillist - using identifiers allow the system to recognize later that it is already a message included in the database.


Do you think of any other?


Regards,
Danyblue

tobiaseigen
Registered User
Posts: 37
Joined: Sun Apr 14, 2002 2:51 pm
Location: Takoma Park, MD USA
Contact:

Re: Basic funcionalities

Post by tobiaseigen » Thu Apr 18, 2002 8:50 am

Hi Danyblue,

Looks like at the moment, we're the only ones interested in this mod - so until we get a phpbb programmer interested, we should probably start with pretty modest desires.

On that note..
danyblue wrote: in my case i think that the following functionalities should be included in the mod.

1 - take mails from the mbox file and gets them into mysql.

I think this is quite doable, either using the phorum model which involves a php script that works with the posting.php script or the mail2mysql model which takes an email, parses it, and dumps the results straight into the database.
danyblue wrote: 2 - Parse the resulting messages against the Phpbb database and look matching subjects.

I don't think this will work - it would make more sense to use the message ID numbers to maintain topic threads. In any case this isn't really the biggest priority for me... the important thing is to get the messages into the database.
danyblue wrote: 3 - Look at sender and search in the user database for this user (probably based on the name of the sender or on the e-mail), if not found automatically register the user.

Clever idea, but this isn't a priority for me either. In my view, we can easily have a mailman mailing list which is maintained separately from the phpbb forums. This makes perfect sense - since not everyone will want to be on the subscriber list but everyone will from time to time want to come and read postings.
danyblue wrote: 4 - If the message is not found on the database, display the message in the screen with a dropdown box containing the forums available, that you can indicate manually where to include the messages that are not recognizer. I am basic this point on the fact that for me a mail list is just like a category where several discussions are taking place and that i want to separte through several forum in order to classify them.

I'm not quite sure what you're driving at here. I agree with you that it would be helpful to maintain topic threads in the phpbb way, but messages have to be posted automatically and immediately into the forums without human intervention.
danyblue wrote: 5 - If a new message is posted directly in the forum, sends it automatically to the corresponding maillist - using identifiers allow the system to recognize later that it is already a message included in the database.

Clearly, we'll have to watch out for "auto-reply wars" in which mail cycles infinitely until the mail server chokes or the hard drive is full.
danyblue wrote: Do you think of any other?

My vision for this mod is for it to be very, very simple. Easy to set up and easy to run - without major modifications to either mailman or phpbb or how people use them. This should be possible with a simple hack of posting.php.

Cheers,

Tobias
Kabissa - Space for change in Africa
http://www.kabissa.org

danyblue
Registered User
Posts: 337
Joined: Fri Apr 12, 2002 4:48 pm
Contact:

Functionalities

Post by danyblue » Thu Apr 18, 2002 9:19 am

Hi tobiaseigen,
My experience is that there is a lot of people jumping into projects once they are already running, initiation them is very difficult and a lot of them just will not reach maturity, so people usually are taking and Wait and See approacha. But i think we can do it.


Based on my message and in yours i think we could define the devolpment into 2 phases:

- Automatic Mode - Take the messages as they are coming and just insert them into mysql database and adjust the the Phpbb configuration.

- Manual Mode - Once we have the first mode running, we will then discuss the other aspects. My main idea here is that the discussion lists have a lot themes going on at the same time, so i would like to have the figure of moderators that will organize those discussion in separate channels in order to have a clear organization of their forums/discussions. I am already implementing this by my self on a site that i have in spanish ( http://opennetworks.rg3.net) and it's when I had the idea to have this thing done the minimum human intervention.


Regarding phase one, any suggestion for work distrubition, it seems that there is already a lot of things done is just a question to see the existing gaps and address them.

Locked

Return to “[2.0.x] MODs in Development”

Who is online

Users browsing this forum: No registered users and 7 guests