[DEV] NewsSync (Usenet -> phpBB Sync)

A place for MOD Authors to post and receive feedback on MODs still in development. No MODs within this forum should be used within a live environment! No new topics are allowed in this forum.
Forum rules
READ: phpBB.com Board-Wide Rules and Regulations

IMPORTANT: MOD Development Forum rules

On February 1, 2009 this forum will be set to read only as part of retiring of phpBB2.
Post Reply
wineknow
Registered User
Posts: 299
Joined: Tue Mar 04, 2003 5:20 pm
Location: Calgary, Alberta, Canada
Contact:

[DEV] NewsSync (Usenet -> phpBB Sync)

Post by wineknow » Tue Mar 04, 2003 5:35 pm

I have decided to split this forum at this point as it is getting quite large.
Please goto topic: NewsSync 1.5 Beta

Please only discuss version 1.0.x in this topic.
thanks,
-k

NewsSync for phpBB
- syncronizes messages between usenet and phpBB

All Available Downloads

version 1.0.4 - released June 5th, 2003
- fixed ordering problem with messages
- updated version number

version 1.0.3 - released June 4th, 2003
- fixes a time conversion bug
- fixes a stupid mistake (by me) that had the name of the database table incorrect :(
- above should also fix the spinning problem reported...

version 1.0.2 - release May 29th, 2003
- fixed empty subject bug (beta bug only)
- updated to support phpNuke version 6.5 with phpBB mod -- to enable: ./newssync_config.php set NUKE_SYNC define to true
- added support for cyrllic character conversion
- cleaned up /CHECKCONFIG for phpNuke stuff - also removed file check for included files
- added support for user signatures added to outgoing messages
- fixed: duplicate messages being added (as well as other odd behavious) if the messageid had a ' in it...


version 1.0 - released April 20, 2003
- intended to run as a UNIX CRON job or WINDOWS scheduled BAT file
- can be run interactively from web browser but command line and cron access a real help
- includes quote mapping (> to phpBB style)
- smart subjects when flattening the tree structure into phpBB flat structure
- multiple newsservers and groups (only 1 per forum)
- requires php version 4.1.x or greater

version 1.0.1 - currently in beta
- support phpNuke version 6.5 with phpBB mod
- added support for cyrllic character conversion
- added support for user signatures added to outgoing messages
- minor bug fixes

All Available Downloads

roadmap future builds

version 1.1
- web-based configuration (moving config files to database)
- optional mod to store alternative emails as part of profile and not in the text file
- installer script (command line or web TBD)
- implement pseudo-cron as part of release
- find windows-based solution to pseudo-cron (TBD)
- enhance NOSPAM to better munge email address on outgoing messages
- remove mailto's through all incoming messages
- and probably more...

version 1.2
- support for next release of phpBB -- depending on timeframe
- attachment support


version 2.0
- TBD

any comments would be appreciated
-k
Last edited by wineknow on Sat Jul 26, 2003 5:31 pm, edited 11 times in total.

netclectic
Former Team Member
Posts: 4439
Joined: Wed Mar 13, 2002 3:08 pm
Location: Omnipresent
Contact:

Post by netclectic » Tue Mar 04, 2003 5:40 pm

Very interesting. Tell us more...

You may also be interested in discussing this with some of the devs here - http://www.m2f.netclectic.com - as various members there have expressed an interesting in usenet / nntp.
m2f wrote: Mail 2 Forum (or M2F) is an add-on software to the phpBB forum system. M2F combines the functionality of a mailing list system and a phpBB forum in order to add bi-directional 'email to forum' or 'forum to email' communication.
Defend the game:
Image

wineknow
Registered User
Posts: 299
Joined: Tue Mar 04, 2003 5:20 pm
Location: Calgary, Alberta, Canada
Contact:

usenet -> phpBB details

Post by wineknow » Tue Mar 04, 2003 6:01 pm

The usenet connection is based off of the Newsportal NNTP<->HTTP Gateway Version: 0.24 by Florian Amrhein <florian.amrhein@web.de> GNU licensed.

My php script runs as a cron job (every hour) and simply retrieves the message headers for comparison with what is in the phpBB forum.
- I keep a map of phpBB post ID with the actual nntp message ID (along with other usenet specific info) stored in a local file (for now)
- when the nntp server expires messages, they get deleted in the phpBB
- if there are any messages in phpBB that my sync program does not know about then they are new posts and need to be sent to the nntp sever as a new message (or reply) -- (working on that now)
- I flatten the nntp message tree so that I only have topic and replies - so its a little crazy trying to verify from a client based newsreader
- I also have done some work to try to detect any binary-only messages and to skip that message
- I have a seperate "hacked" version of functions_post.php (only used with my sync program - where I had to make two changes:
1) replace include with include_once (so I can post over and over)
2) change to the current_time so that I could post based on the usenet message time.

So in essence this is a traditional hack gluing together two sets of code the phpBB and the news portal which neither of them I understand too well (especially the latter).

-k

netclectic
Former Team Member
Posts: 4439
Joined: Wed Mar 13, 2002 3:08 pm
Location: Omnipresent
Contact:

Post by netclectic » Tue Mar 04, 2003 7:06 pm

Excellent, sounds very intersting. Is there anywhere i can look at what you have (code) so far?

As far as the functions_post thing goes, you might be interested in some functions i've put together - [FUNCTION] Insert Post
This functions is used to insert a post into your phpbb forums. It handles all the related bits like updating post counts, indexing search words, etc. The post is inserted for a specific user, so you will have to already have a user setup which you want to use with it.
Defend the game:
Image

kolczyk
Registered User
Posts: 73
Joined: Sat Dec 14, 2002 8:57 pm

Post by kolczyk » Wed Mar 05, 2003 11:07 am

hi,

I'm very interesting!
please post a mod!

president
Registered User
Posts: 169
Joined: Thu Apr 04, 2002 11:31 pm

Post by president » Wed Mar 05, 2003 1:33 pm

Yes I'm also very interested.

I know that in Ireland alot of College Internet Societys Have a shared nntp feed, it would be cool to get phpBB2 tied into it :)

wineknow
Registered User
Posts: 299
Joined: Tue Mar 04, 2003 5:20 pm
Location: Calgary, Alberta, Canada
Contact:

will release it this weekend BETA1

Post by wineknow » Wed Mar 05, 2003 3:19 pm

I am cleaning up the code in my free time.
Will post a beta this weekend.

btw: in limited testing (to alt.test) I do have replys and new topics from phpBB being posted to the newsgroup...

(don't worry lots-o-bugs still ;)

-kevin

cymru2000
Registered User
Posts: 33
Joined: Sat Apr 06, 2002 1:27 am
Location: London
Contact:

Post by cymru2000 » Wed Mar 05, 2003 11:37 pm

would absolutly love to see this one come through :D

wineknow
Registered User
Posts: 299
Joined: Tue Mar 04, 2003 5:20 pm
Location: Calgary, Alberta, Canada
Contact:

Alpha 1 of newssync posted

Post by wineknow » Fri Mar 07, 2003 1:03 am

You can download alpha 1 of newssync at:
www.wineknow.com/downloads/newssync_a1.tar

If you don't know what you are doing, then don't use this.
Only tried on Linux - Windows should work.

Under no circumstances do NOT enable message replies on anything but a .test newsgroup.

Readme's are currently at the top of newssync.php and the config file.

Let me know what I missed in this tar.

Please send comments.
-kevin

president
Registered User
Posts: 169
Joined: Thu Apr 04, 2002 11:31 pm

Post by president » Fri Mar 07, 2003 4:34 am

I don't quiet understand how i'm supposed to install this.

jlin
Registered User
Posts: 26
Joined: Sat Feb 01, 2003 5:25 pm

works great!!!

Post by jlin » Fri Mar 07, 2003 5:25 am

works great!!!

See the demo here http://www.itaiwan.org/viewforum.php?f=40

I limit the posts to 30 by editing the config file.

User avatar
Pda0
Registered User
Posts: 217
Joined: Sun Apr 14, 2002 5:00 am
Location: Chile

Post by Pda0 » Fri Mar 07, 2003 11:52 am

wineknow: Are you interested in helping us write the nntp plug-in for M2F? I guess you just have what it takes :)

(http://www.m2f.netclectic.com)
.pd

jlin
Registered User
Posts: 26
Joined: Sat Feb 01, 2003 5:25 pm

Post by jlin » Fri Mar 07, 2003 2:02 pm

how do you set up cron job on cpanel? my newsgroup isn't updating automatically.

1) does 0 0 * * * mean every day at midnight?

2) do i find the path using phpinfo(), i.e. /home/mydomain/public_html/nntp/newssync.php

thanks

wineknow
Registered User
Posts: 299
Joined: Tue Mar 04, 2003 5:20 pm
Location: Calgary, Alberta, Canada
Contact:

install help and a little bit on cron

Post by wineknow » Fri Mar 07, 2003 2:41 pm

ok, good -- confusion on install would mean that folks are trying to use it and it's not quite cooked yet.

to install:
on Linux
- untar the file into a subdir of phpBB (for this alpha)
- change permissions: chown -R [user] nameofsubdir
(where user is who you want to run it as)
- in phpBB create a new forum eg alt.test
- go into permissions -> advanced mode and set this to be view/read for ALL and ADMIN for the rest
- view the new form - in the URL there will be a f=[n] -- remember it
- edit the newssync_config.php file
- change $newsSync_newsGroupList = array(n => "alt.test") and replace the n with the number you were suppose to remember above
-- DEV NOTE: yes this is crap -- I plan to rewrite this -- where there will be an external text file and you will list phpBB forum name, newsgroup name -- newsync will find the forumID itself...
- change the $server= parameter to the name of your news server
-- if your ISP makes you log-in, then you will need to do a few more things in that section.
- save the file.

edit the gosync file
change the directory to your absolute directory (needs to be absolute if you are going to run it from cron)
save the file
chmod +x gosync -- make it an executable

note: the included PHP.INI file is ony there for ONE thing: I have the timeout set to something ridiclously long -- otherwise php stops executing after 30 seconds by default -- I don't recommend changing your real php.ini (in /etc) because you can end up hanging your webserver (or giving that appearance) as rouge processes could run for a long time...

now hold your breath - cross your fingers -- whatever works for you.
run gosync (./gosync)

In theory, it should start adding stuff to your phpBB

* if it doesn't, and you can't figure it out post the first error right here...

I don't recommend setting up a cron until you are kinda sure that things are actually working....

a note about cron:
- be running as the user you want to run the script as (root is ok - not should be your last choice)
- crontab -e
- [you are now in vi]
- to run every hour: 0 * * * * /absolute/path/to/the/gosync 1>dev/null 2>/dev/null
- ZZ to exit crontab edit

run crontab -l
you should see your command
(note: the 1>dev/null stuff is so you don't get ANY messages on the console)

jlin
Registered User
Posts: 26
Joined: Sat Feb 01, 2003 5:25 pm

Post by jlin » Fri Mar 07, 2003 3:54 pm

if we dont have shell access can we just do this instead?
1) does 0 0 * * * mean every day at midnight?

2) do i find the path using phpinfo(), i.e. /home/mydomain/public_html/nntp/newssync.php

Post Reply

Return to “[2.0.x] MODs in Development”