[DEV] NewsSync (Usenet -> phpBB Sync)

A place for MOD Authors to post and receive feedback on MODs still in development. No MODs within this forum should be used within a live environment! No new topics are allowed in this forum.
Forum rules
READ: phpBB.com Board-Wide Rules and Regulations

IMPORTANT: MOD Development Forum rules

On February 1, 2009 this forum will be set to read only as part of retiring of phpBB2.
Post Reply
Legolas-ga
Registered User
Posts: 115
Joined: Tue Jan 07, 2003 8:39 am
Location: Costa Mesa, CA - but, moving to Corona, CA
Contact:

Post by Legolas-ga » Sat Mar 15, 2003 3:36 am

oh yeah... PLEASE PLEASE PLEASE have the script skip over posts that are 'announcements' in the usenet forums. That will give me a way to explain things to my users--without it being posted on the usenet.. and of course, ignore the announcement message for expiry, purging, etc..

Thanks!

Legolas-ga
Registered User
Posts: 115
Joined: Tue Jan 07, 2003 8:39 am
Location: Costa Mesa, CA - but, moving to Corona, CA
Contact:

Post by Legolas-ga » Sat Mar 15, 2003 6:49 am

hm.... output.log seems to be overwritten each time newssync.php is run.. Previous version had the log appended.. I'm thinking there's a bug there ;)

kolczyk
Registered User
Posts: 73
Joined: Sat Dec 14, 2002 8:57 pm

Post by kolczyk » Sat Mar 15, 2003 11:33 am

hey it's working fine now!
a lot of time I waste for look fast news server with posting allowing....
Thanks!

Legolas-ga
Registered User
Posts: 115
Joined: Tue Jan 07, 2003 8:39 am
Location: Costa Mesa, CA - but, moving to Corona, CA
Contact:

Post by Legolas-ga » Sat Mar 15, 2003 4:45 pm

care to share the fast/good server that allows posting? :)

kolczyk
Registered User
Posts: 73
Joined: Sat Dec 14, 2002 8:57 pm

Post by kolczyk » Sun Mar 16, 2003 7:31 am

there is one small bug... when someone reply to the same topic (on usenet and phpBB) before ressync new thread is created on usenet

so I think if someone reply in usenet before resync and that reply wasn't available at phpBB, phpBB will be created another thread with phpBB replies :(

wineknow
Registered User
Posts: 299
Joined: Tue Mar 04, 2003 5:20 pm
Location: Calgary, Alberta, Canada
Contact:

moving towards 0.8 beta

Post by wineknow » Sun Mar 16, 2003 2:50 pm

kolczyk wrote: there is one small bug... when someone reply to the same topic (on usenet and phpBB) before ressync new thread is created on usenet

so I think if someone reply in usenet before resync and that reply wasn't available at phpBB, phpBB will be created another thread with phpBB replies :(


OK - 0.8 I want to be feature complete, so here are the features/bug fixes I hope to implement
- the one above in the quote
- fix bug that deleted the logfile at beginning of every run (already fixed that)
TODO - skip over posts that are announcements, ...
look into - expiry time OPTION to only delete in newssync table and let phpBB purge the data
TODO - get rid of the rest of the codes color/bold/italics/....

I hope to have this version about this week, so then the only thing left is to better document INSTALL and fix critical bugs.

-k

wineknow
Registered User
Posts: 299
Joined: Tue Mar 04, 2003 5:20 pm
Location: Calgary, Alberta, Canada
Contact:

anyone good at regular expression?

Post by wineknow » Sun Mar 16, 2003 6:29 pm

So I am trying to write easy code to do the following

change:
[i:5014e01d04]italics text[/i:5014e01d04]
into:
italics text

here is my code:
$bbcode = "i";
$newmsg = "[i:5014e01d04]italics text[/i:5014e01d04]";
$newmsg = preg_replace("/\[$bbcode^(\])*\]/","",$newmsg);
$newmsg = preg_replace("/\[\/$bbcode^(\])*\]/","",$newmsg);
- which does not work...

any ideas - before I go and lookup regex info elsewhere?
-k

Legolas-ga
Registered User
Posts: 115
Joined: Tue Jan 07, 2003 8:39 am
Location: Costa Mesa, CA - but, moving to Corona, CA
Contact:

Post by Legolas-ga » Sun Mar 16, 2003 6:37 pm

Couldn't you reuse the bbcode functions from phpbb's bbcode functions? :)

wineknow
Registered User
Posts: 299
Joined: Tue Mar 04, 2003 5:20 pm
Location: Calgary, Alberta, Canada
Contact:

working towards a 0.8beta

Post by wineknow » Sun Mar 16, 2003 11:39 pm

Just that I would let you know where newsSync is at:
- fixed the logging bug
- by default only "normal topics" are send to the news server and Announcements and Stickys are not -- it is configurable if you don't like that
- remove all the BBCodes from a message before it is sent to nntp
- now do some checking to make sure that the memory_size is set as opposed to just setting it and assuming it works

so I am pretty sure this is feature complete.
- the outstanding big bug is the issue that sometimes messages from usenet become topics and not replies -- which is a little narley to fix because it was NOT my code that setup most of it
- so once I track that down, I will release B8 that should work for most folks

TWO things I can't (or not going to do for 1.0) is the following:
- newsgroups on different nntp servers
--- you can run two copies of newsync in two different subdirs to get around that issue...

- allow phpBB to autoprune this forum instead of keeping the nntp server in sync with phpBB. This is possible but is a bit of message code and has the potential to grow the newssync db table very large...so I am going to skip it.


-k

brasco
Registered User
Posts: 28
Joined: Tue Apr 23, 2002 12:22 am
Contact:

Post by brasco » Mon Mar 17, 2003 3:16 am

Made sure I read the docs on this one as didnt want to end of spamming usenet etc :)

Really nice mod, excellent idea, seen it done with vbulletin before but not with phpBB.

Was gutted though as when it didnt work, thought it test connectivity by tellnetting to the newsgroup server, it failed instantly.

Turns out the ISP has a deny all policy outgoing and only allow a few things, probably just ftp/http maybe mysql etc. Will test this on another box as soon as I can.

Can you actually set a limit of how many posts are pulled down, or the last x days, or max size etc. etc. Not bothered about binaries would be text only ng's just thinking about database sizes for larger newsgroups

Nice work
Image
--------------------------------------------
http://www.security-forums.com

wineknow
Registered User
Posts: 299
Joined: Tue Mar 04, 2003 5:20 pm
Location: Calgary, Alberta, Canada
Contact:

Post by wineknow » Mon Mar 17, 2003 3:38 am

brasco wrote: Made sure I read the docs on this one as didnt want to end of spamming usenet etc :)

Really nice mod, excellent idea, seen it done with vbulletin before but not with phpBB.

Was gutted though as when it didnt work, thought it test connectivity by tellnetting to the newsgroup server, it failed instantly.

Turns out the ISP has a deny all policy outgoing and only allow a few things, probably just ftp/http maybe mysql etc. Will test this on another box as soon as I can.

Can you actually set a limit of how many posts are pulled down, or the last x days, or max size etc. etc. Not bothered about binaries would be text only ng's just thinking about database sizes for larger newsgroups

Nice work


- thanks for the feedback.
- I have found that it is a problem with many ISP's they really limit how (and if) you can access their news server
- If you are brave and have a spare old computer with a big hard disk, you can setup your own NNTP server, where you can define your own rules on who and how you can connect to it...that said, I have never done it...
- I too wanted to be very careful NOT to spam any newsgroups (even .test ones)
- there are many flags and switches including the ability to limit the message size -- regardless of content type
- I try to weed out as much binary "goo" as possible, but I am still looking for a more solid way. Some binary stuff still gets in...
- Messages are expired at the same time the NNTP server expires them, so unless you change the file size limiter you won't get a massive database...
-k

wineknow
Registered User
Posts: 299
Joined: Tue Mar 04, 2003 5:20 pm
Location: Calgary, Alberta, Canada
Contact:

newssync 0.8 beta posted

Post by wineknow » Tue Mar 18, 2003 3:30 pm

Hi folks,

changes:
- fixed bug that deleted the logfile at beginning of every run (oops ;)
- skip over TOPICS that are announcements and stickys -> by default only allow "normal posts"
- remove bbcodes ([URL],, ...) within the phpBB message when sending out to usenet
- fixed an error in sorting, where messages were not sorted correctly, which would at times cause replies to think they are topics...
- added some checks to make sure that the memory size is being set right (i.e. safe mode is disabled)
- made a fix in the message time conversion, so that in case we can't convert it properly OR the user's clock is set wrong, no messages will get posts with date/time in the future.

download at:
http://www.wineknow.com/downloads/newssync_0.8_beta.tar

let me know if you have any problems.
-k

User avatar
Mary at Coupon Forum
Registered User
Posts: 71
Joined: Sat Sep 14, 2002 7:55 am
Contact:

Post by Mary at Coupon Forum » Tue Mar 18, 2003 6:45 pm

Awesome! Been wanting this!
Mary at Coupon Forum
I hate penguins. :D

dojohn
Registered User
Posts: 11
Joined: Tue Mar 18, 2003 3:37 am

Re: newssync 0.8 beta posted

Post by dojohn » Wed Mar 19, 2003 8:40 am

wineknow wrote: download at:
http://www.wineknow.com/downloads/newssync_0.8_beta.tar

let me know if you have any problems.

Dear kevin:

I try the 0.8 beta...but the problem still have .

1. The reply post always new a thread. :cry:

2. When sync...I always stop at one thread...the error message is :
Error in SQL: INSERT INTO phpbb_posts (topic_id, forum_id, poster_id,
post_username, post_time, poster_ip, enable_bbcode, enable_html,
enable_smilies, enable_sig) VALUES (1493, 6, -1, '/_\', 1047739920,
'00000000', 1, 1, 0, 0)
.
I think you will fix this in the 0.9 beta. 8)

My forum is in :
http://dojohn.adsldns.org/phpBB2/viewforum.php?f=6

PS. I'm from Taiwan...so the charset is set to be zh_TW.Big5

wineknow
Registered User
Posts: 299
Joined: Tue Mar 04, 2003 5:20 pm
Location: Calgary, Alberta, Canada
Contact:

Re: newssync 0.8 beta posted

Post by wineknow » Wed Mar 19, 2003 3:00 pm

dojohn wrote: PS. I'm from Taiwan...so the charset is set to be zh_TW.Big5


This is a problem.
I have not done any testing for multi-byte languages. I do not have the necessary skills to debug/code the third party nntp/php software I found to get it to work with multibyte, let alone understand how mySQL and my stuff needs to interact.

Unfortunately, newssync v1.0 will not support multibyte languages...

sorry,
-kevin

Post Reply

Return to “[2.0.x] MODs in Development”