Page 1 of 38

[DEV] NewsSync (Usenet -> phpBB Sync)

Posted: Tue Mar 04, 2003 5:35 pm
by wineknow
I have decided to split this forum at this point as it is getting quite large.
Please goto topic: NewsSync 1.5 Beta

Please only discuss version 1.0.x in this topic.
thanks,
-k

NewsSync for phpBB
- syncronizes messages between usenet and phpBB

All Available Downloads

version 1.0.4 - released June 5th, 2003
- fixed ordering problem with messages
- updated version number

version 1.0.3 - released June 4th, 2003
- fixes a time conversion bug
- fixes a stupid mistake (by me) that had the name of the database table incorrect :(
- above should also fix the spinning problem reported...

version 1.0.2 - release May 29th, 2003
- fixed empty subject bug (beta bug only)
- updated to support phpNuke version 6.5 with phpBB mod -- to enable: ./newssync_config.php set NUKE_SYNC define to true
- added support for cyrllic character conversion
- cleaned up /CHECKCONFIG for phpNuke stuff - also removed file check for included files
- added support for user signatures added to outgoing messages
- fixed: duplicate messages being added (as well as other odd behavious) if the messageid had a ' in it...


version 1.0 - released April 20, 2003
- intended to run as a UNIX CRON job or WINDOWS scheduled BAT file
- can be run interactively from web browser but command line and cron access a real help
- includes quote mapping (> to phpBB style)
- smart subjects when flattening the tree structure into phpBB flat structure
- multiple newsservers and groups (only 1 per forum)
- requires php version 4.1.x or greater

version 1.0.1 - currently in beta
- support phpNuke version 6.5 with phpBB mod
- added support for cyrllic character conversion
- added support for user signatures added to outgoing messages
- minor bug fixes

All Available Downloads

roadmap future builds

version 1.1
- web-based configuration (moving config files to database)
- optional mod to store alternative emails as part of profile and not in the text file
- installer script (command line or web TBD)
- implement pseudo-cron as part of release
- find windows-based solution to pseudo-cron (TBD)
- enhance NOSPAM to better munge email address on outgoing messages
- remove mailto's through all incoming messages
- and probably more...

version 1.2
- support for next release of phpBB -- depending on timeframe
- attachment support


version 2.0
- TBD

any comments would be appreciated
-k

Posted: Tue Mar 04, 2003 5:40 pm
by netclectic
Very interesting. Tell us more...

You may also be interested in discussing this with some of the devs here - http://www.m2f.netclectic.com - as various members there have expressed an interesting in usenet / nntp.
m2f wrote: Mail 2 Forum (or M2F) is an add-on software to the phpBB forum system. M2F combines the functionality of a mailing list system and a phpBB forum in order to add bi-directional 'email to forum' or 'forum to email' communication.

usenet -> phpBB details

Posted: Tue Mar 04, 2003 6:01 pm
by wineknow
The usenet connection is based off of the Newsportal NNTP<->HTTP Gateway Version: 0.24 by Florian Amrhein <florian.amrhein@web.de> GNU licensed.

My php script runs as a cron job (every hour) and simply retrieves the message headers for comparison with what is in the phpBB forum.
- I keep a map of phpBB post ID with the actual nntp message ID (along with other usenet specific info) stored in a local file (for now)
- when the nntp server expires messages, they get deleted in the phpBB
- if there are any messages in phpBB that my sync program does not know about then they are new posts and need to be sent to the nntp sever as a new message (or reply) -- (working on that now)
- I flatten the nntp message tree so that I only have topic and replies - so its a little crazy trying to verify from a client based newsreader
- I also have done some work to try to detect any binary-only messages and to skip that message
- I have a seperate "hacked" version of functions_post.php (only used with my sync program - where I had to make two changes:
1) replace include with include_once (so I can post over and over)
2) change to the current_time so that I could post based on the usenet message time.

So in essence this is a traditional hack gluing together two sets of code the phpBB and the news portal which neither of them I understand too well (especially the latter).

-k

Posted: Tue Mar 04, 2003 7:06 pm
by netclectic
Excellent, sounds very intersting. Is there anywhere i can look at what you have (code) so far?

As far as the functions_post thing goes, you might be interested in some functions i've put together - [FUNCTION] Insert Post
This functions is used to insert a post into your phpbb forums. It handles all the related bits like updating post counts, indexing search words, etc. The post is inserted for a specific user, so you will have to already have a user setup which you want to use with it.

Posted: Wed Mar 05, 2003 11:07 am
by kolczyk
hi,

I'm very interesting!
please post a mod!

Posted: Wed Mar 05, 2003 1:33 pm
by president
Yes I'm also very interested.

I know that in Ireland alot of College Internet Societys Have a shared nntp feed, it would be cool to get phpBB2 tied into it :)

will release it this weekend BETA1

Posted: Wed Mar 05, 2003 3:19 pm
by wineknow
I am cleaning up the code in my free time.
Will post a beta this weekend.

btw: in limited testing (to alt.test) I do have replys and new topics from phpBB being posted to the newsgroup...

(don't worry lots-o-bugs still ;)

-kevin

Posted: Wed Mar 05, 2003 11:37 pm
by cymru2000
would absolutly love to see this one come through :D

Alpha 1 of newssync posted

Posted: Fri Mar 07, 2003 1:03 am
by wineknow
You can download alpha 1 of newssync at:
www.wineknow.com/downloads/newssync_a1.tar

If you don't know what you are doing, then don't use this.
Only tried on Linux - Windows should work.

Under no circumstances do NOT enable message replies on anything but a .test newsgroup.

Readme's are currently at the top of newssync.php and the config file.

Let me know what I missed in this tar.

Please send comments.
-kevin

Posted: Fri Mar 07, 2003 4:34 am
by president
I don't quiet understand how i'm supposed to install this.

works great!!!

Posted: Fri Mar 07, 2003 5:25 am
by jlin
works great!!!

See the demo here http://www.itaiwan.org/viewforum.php?f=40

I limit the posts to 30 by editing the config file.

Posted: Fri Mar 07, 2003 11:52 am
by Pda0
wineknow: Are you interested in helping us write the nntp plug-in for M2F? I guess you just have what it takes :)

(http://www.m2f.netclectic.com)
.pd

Posted: Fri Mar 07, 2003 2:02 pm
by jlin
how do you set up cron job on cpanel? my newsgroup isn't updating automatically.

1) does 0 0 * * * mean every day at midnight?

2) do i find the path using phpinfo(), i.e. /home/mydomain/public_html/nntp/newssync.php

thanks

install help and a little bit on cron

Posted: Fri Mar 07, 2003 2:41 pm
by wineknow
ok, good -- confusion on install would mean that folks are trying to use it and it's not quite cooked yet.

to install:
on Linux
- untar the file into a subdir of phpBB (for this alpha)
- change permissions: chown -R [user] nameofsubdir
(where user is who you want to run it as)
- in phpBB create a new forum eg alt.test
- go into permissions -> advanced mode and set this to be view/read for ALL and ADMIN for the rest
- view the new form - in the URL there will be a f=[n] -- remember it
- edit the newssync_config.php file
- change $newsSync_newsGroupList = array(n => "alt.test") and replace the n with the number you were suppose to remember above
-- DEV NOTE: yes this is crap -- I plan to rewrite this -- where there will be an external text file and you will list phpBB forum name, newsgroup name -- newsync will find the forumID itself...
- change the $server= parameter to the name of your news server
-- if your ISP makes you log-in, then you will need to do a few more things in that section.
- save the file.

edit the gosync file
change the directory to your absolute directory (needs to be absolute if you are going to run it from cron)
save the file
chmod +x gosync -- make it an executable

note: the included PHP.INI file is ony there for ONE thing: I have the timeout set to something ridiclously long -- otherwise php stops executing after 30 seconds by default -- I don't recommend changing your real php.ini (in /etc) because you can end up hanging your webserver (or giving that appearance) as rouge processes could run for a long time...

now hold your breath - cross your fingers -- whatever works for you.
run gosync (./gosync)

In theory, it should start adding stuff to your phpBB

* if it doesn't, and you can't figure it out post the first error right here...

I don't recommend setting up a cron until you are kinda sure that things are actually working....

a note about cron:
- be running as the user you want to run the script as (root is ok - not should be your last choice)
- crontab -e
- [you are now in vi]
- to run every hour: 0 * * * * /absolute/path/to/the/gosync 1>dev/null 2>/dev/null
- ZZ to exit crontab edit

run crontab -l
you should see your command
(note: the 1>dev/null stuff is so you don't get ANY messages on the console)

Posted: Fri Mar 07, 2003 3:54 pm
by jlin
if we dont have shell access can we just do this instead?
1) does 0 0 * * * mean every day at midnight?

2) do i find the path using phpinfo(), i.e. /home/mydomain/public_html/nntp/newssync.php