[DEV] NewsSync (Usenet -> phpBB Sync)

A place for MOD Authors to post and receive feedback on MODs still in development. No MODs within this forum should be used within a live environment! No new topics are allowed in this forum.
Forum rules
READ: phpBB.com Board-Wide Rules and Regulations

IMPORTANT: MOD Development Forum rules

On February 1, 2009 this forum will be set to read only as part of retiring of phpBB2.
wineknow
Registered User
Posts: 299
Joined: Tue Mar 04, 2003 5:20 pm
Location: Calgary, Alberta, Canada
Contact:

Post by wineknow »

glad to hear that you got things working.

as for the password security question. nntp sends userid/passwds in clear text so no, your password is not secure.

the password is stored in the newssync-grouplist.txt file, so it is as secure as say, phpbb or phpmysql in that respect...

for various newsgroups I use http://news.cis.dfn.de/ as my newsserver.
You have to request a user/id and password and they will approve and assign you one. The passwd is not secure but at least is is not your ISP password.

hope this helps.
-k


iRuben wrote:
wineknow wrote:
iRuben wrote:...I constantly get an Error sending message: 500 What?...


this is odd, can you PM me the /CHECKCONFIG results as well as the pertiant log information. This was the least amount and complicated code in the whole program...Can you send messages to that newsserver via news client?


Thank for wanting to help, but it's already solved. Don't know exactly what was wrong (yes, I WAS able to post using a "normal" news client), but I solved it using a "official" newsserver with a username/password. My own provider wouldn't allow this (I need to be on their cable connection and the provider's server isn't), but I use my girlfriends ADSL provider, she has a dial-in account there as well, so they accept connections from anywhere as long as your provide a username/password. :-D

My only concern now is; how safe is the username/password info in the config file that is now on my providers server?
wineknow
Registered User
Posts: 299
Joined: Tue Mar 04, 2003 5:20 pm
Location: Calgary, Alberta, Canada
Contact:

version 1.0.4 now available

Post by wineknow »

the message sorting issue is now fixed
it was the problem outlined above
thanks for your testing qnx

see first message in this topic for download details
-k
qnx
Registered User
Posts: 69
Joined: Fri Mar 28, 2003 6:03 am

Post by qnx »

it seems the SubjectCompare is not working. I seems to remember it used to work before 1.0.4.
check this http://test.openqnx.com/forum/viewforum.php?f=3
some of those should be in the same thread (same subject), but not.

just tried to revert back to 1.0.3, and confirmed working.
Last edited by qnx on Fri Jun 06, 2003 9:06 pm, edited 1 time in total.
qnx
Registered User
Posts: 69
Joined: Fri Mar 28, 2003 6:03 am

Post by qnx »

the number of "Posts" in the phpbb2 Index page doesn't match the number of articles on the news server after the initial import. I guess it is caused by those "Unfortunately, this message is unavailable." posts? If you look closely, it is available, just newssync created a new thread for that message. Can't we use the "compare subject" technique to eliminate those "Unfortunately, this message is unavailable." problems?

Just tried 1.0.3 and there is no such problem. looks like "compare subject" quit working in 1.0.4.
qnx
Registered User
Posts: 69
Joined: Fri Mar 28, 2003 6:03 am

performance

Post by qnx »

I am running "php" from command line/

I did an initial import of a group with about 11000 messages, which take over 2 hours.

then I did a "update" run:

Syncing: usenet:qnn.os with phpBB:qnn.os
-- approximately 10971 messages (pauses about 5 seconds)
* Looking for new messages... (it paused about 20 seconds)
-- no new topics or replys
* Looking for new phpBB posts... (it paused about anoth 25 seconds).
-- no new phpBB posts

** Update Complete **
now it is taking even longer and I got the following error
<br>
<b>Fatal error</b>: Maximum execution time of 30 seconds exceeded in <b>Unknown</b> on line <b>0</b><br>

and at the time of my typing this post, it is still going.

two questions:
1) apparently, we need to fix that "Fatal error".
2) are there ways to import the performance? I can understand the long time during the initial import (maybe there are ways to speed that also?), but the subsequent "update" runs should be faster. My guess is it is doing database search. If so, maybe proper "indexing" will help?

thanks!
wineknow
Registered User
Posts: 299
Joined: Tue Mar 04, 2003 5:20 pm
Location: Calgary, Alberta, Canada
Contact:

Re: performance

Post by wineknow »

qnx wrote: I am running "php" from command line/

I did an initial import of a group with about 11000 messages, which take over 2 hours.

then I did a "update" run:

Syncing: usenet:qnn.os with phpBB:qnn.os
-- approximately 10971 messages (pauses about 5 seconds)
* Looking for new messages... (it paused about 20 seconds)
-- no new topics or replys
* Looking for new phpBB posts... (it paused about anoth 25 seconds).
-- no new phpBB posts

** Update Complete **
now it is taking even longer and I got the following error
<br>
<b>Fatal error</b>: Maximum execution time of 30 seconds exceeded in <b>Unknown</b> on line <b>0</b><br>

and at the time of my typing this post, it is still going.

two questions:
1) apparently, we need to fix that "Fatal error".
2) are there ways to import the performance? I can understand the long time during the initial import (maybe there are ways to speed that also?), but the subsequent "update" runs should be faster. My guess is it is doing database search. If so, maybe proper "indexing" will help?

thanks!


the fatal error is a php issue. there is a 30 second timeout set which I try to override (if the configuration/version) of php allows it...

when you run the /CHECKCONFIG does it do any complaining about can't override time?

I an not an expert in SQL (and don't really plan to learn), so if there is anyone who can tell me that I am not indexing a field, or whatever to speed things up, that would be greatly appreciated.

that said, the looking for new posts sql is ineffecient as hell, I will add it to my list of stuff to address for the next release.

I do know there is other users with newsgroups as large as 20,000 messages; and you are correct where the first sync takes a while, while future syncs do not...

while newssync is running, check to make sure the machine is not running low (or out) of memory -- always look to the easy fix first ;)

-k
fshagan
Registered User
Posts: 169
Joined: Mon Dec 09, 2002 1:41 am
Location: California
Contact:

Filters for Cross Posts, etc.

Post by fshagan »

Wineknow, the "punishment" for making a great mod is that everyone will start pestering you with worthless little improvements you can make to it. Like I'm about to do :wink:

Have you considered a way to add a filter to the mod? Or make it so it can call something like nfilter (http://www.nfilter.org, which is getting praise all around? Its OpenSource, so I suppose you could just extract some code from it and give credit to the original authors.

Manually moderating the forum is pretty time consuming; I'd love to be able to zap cross-posted messages, political diatribes, etc. before importing them.
qnx
Registered User
Posts: 69
Joined: Fri Mar 28, 2003 6:03 am

Re: performance

Post by qnx »

wineknow wrote: the fatal error is a php issue. there is a 30 second timeout set which I try to override (if the configuration/version) of php allows it...

when you run the /CHECKCONFIG does it do any complaining about can't override time?

while newssync is running, check to make sure the machine is not running low (or out) of memory -- always look to the easy fix first ;)

-k


the first run took more than two hours and it didn't timeout, so I don't know there is a problem with 30 second timeout.

php newssync.php /CHECKCONFIG
X-Powered-By: PHP/4.1.2
Content-type: text/html



NewsSync for phpBB version 1.0.4
(c) copyright 2003 wineknow.com

-- Current PHP version: 4.1.2
-- phpBB version 2.0.4
-- Local time offset for phpBB is 0 GMT
-- Using Interface Type: cgi
-- Checking memory override limit to 100M ...ok
-- Checking max_execution_time override...ok

...

things look fine to me.

I don't know why the subsequent "update" run will timeout and gave that error. BTW, it hang there forever yesterday and I killed it after several hours.

I did have checked the cpu/mem during the run, and there are no issues.

any ideas?
qnx
Registered User
Posts: 69
Joined: Fri Mar 28, 2003 6:03 am

Re: performance

Post by qnx »

wineknow wrote: I do know there is other users with newsgroups as large as 20,000 messages; and you are correct where the first sync takes a while, while future syncs do not...
-k


I can live with the initial import slownness (you only need to suffer this once anyway :), but my problem is the "future syncs" is slow and gives the 30 seconds timeout errors.
1) why would the "update" sync takes more than 30 seconds?
2) even so, why would it "timeout" while the initial import didn't?

thanks!
qnx
Registered User
Posts: 69
Joined: Fri Mar 28, 2003 6:03 am

Re: performance

Post by qnx »

qnx wrote: Syncing: usenet:qnn.os with phpBB:qnn.os
-- approximately 10971 messages (pauses about 5 seconds)
* Looking for new messages... (it paused about 20 seconds)
-- no new topics or replys
* Looking for new phpBB posts... (it paused about anoth 25 seconds).
-- no new phpBB posts

** Update Complete **
now it is taking even longer and I got the following error
<br>
<b>Fatal error</b>: Maximum execution time of 30 seconds exceeded in <b>Unknown</b> on line <b>0</b><br>


You can see that the "hang" happens AFTER the message "Update Complete", what is "newssync" doing at that time?
Also if you look at the time I gave, it passed 30 seconds on the first few steps already, way before the "Update Complete"...
mood
Registered User
Posts: 18
Joined: Tue Apr 15, 2003 11:20 am
Location: Zurich-Switzerland

Re: performance

Post by mood »

qnx wrote:
qnx wrote:Syncing: usenet:qnn.os with phpBB:qnn.os
-- approximately 10971 messages (pauses about 5 seconds)
* Looking for new messages... (it paused about 20 seconds)
-- no new topics or replys
* Looking for new phpBB posts... (it paused about anoth 25 seconds).
-- no new phpBB posts

** Update Complete **
now it is taking even longer and I got the following error
<br>
<b>Fatal error</b>: Maximum execution time of 30 seconds exceeded in <b>Unknown</b> on line <b>0</b><br>


You can see that the "hang" happens AFTER the message "Update Complete", what is "newssync" doing at that time?
Also if you look at the time I gave, it passed 30 seconds on the first few steps already, way before the "Update Complete"...


Could it be that the news server is causing this? I'm using a local Dnews box, Newsfeeds.com and Freshmeat (announce) and none produce this effect.

Have you tried capturing packets to see what's happening? Could be a million things and I doubt newssync would produce this sort of effect on its own.

Mood
qnx
Registered User
Posts: 69
Joined: Fri Mar 28, 2003 6:03 am

Re: performance

Post by qnx »

mood wrote: Could it be that the news server is causing this? I'm using a local Dnews box, Newsfeeds.com and Freshmeat (announce) and none produce this effect.

Have you tried capturing packets to see what's happening? Could be a million things and I doubt newssync would produce this sort of effect on its own.

Mood


the news server is on localhost (inn2.3.2), and it is firewall protected only serving localhost, I am the only one using this test server and it works fine for the thounds of articles during the initial imports. I can't imagine why it would fail on the "update" run where there is no new articles. Besides, if you read the message I posted, it should have passed the stage where newssync talks to news server.
wineknow
Registered User
Posts: 299
Joined: Tue Mar 04, 2003 5:20 pm
Location: Calgary, Alberta, Canada
Contact:

Re: performance

Post by wineknow »

qnx wrote:
qnx wrote:Syncing: usenet:qnn.os with phpBB:qnn.os
-- approximately 10971 messages (pauses about 5 seconds)
* Looking for new messages... (it paused about 20 seconds)
-- no new topics or replys
* Looking for new phpBB posts... (it paused about anoth 25 seconds).
-- no new phpBB posts

** Update Complete **
now it is taking even longer and I got the following error
<br>
<b>Fatal error</b>: Maximum execution time of 30 seconds exceeded in <b>Unknown</b> on line <b>0</b><br>


You can see that the "hang" happens AFTER the message "Update Complete", what is "newssync" doing at that time?
Also if you look at the time I gave, it passed 30 seconds on the first few steps already, way before the "Update Complete"...


after the "Update Complete" message
newssync -- closes the logfile, deletes the .lock file and then calls exit;

to track down where this bizare behaviour is occuring, edit the inc/sync_utils.php file
the first function (sync_ExitGracefully)
change:

Code: Select all

	LogItClose();
	unlink($newsSync_lockFile);
to

Code: Select all

       echo "about to close logfile\n";
	LogItClose();
       echo "closed logfile: about to delete .lock file\n";
	unlink($newsSync_lockFile);
       echo "deleted lock file about to call exit\n";
letmeknow.
-k
qnx
Registered User
Posts: 69
Joined: Fri Mar 28, 2003 6:03 am

Re: performance

Post by qnx »

wineknow wrote: after the "Update Complete" message
newssync -- closes the logfile, deletes the .lock file and then calls exit;

to track down where this bizare behaviour is occuring, edit the inc/sync_utils.php file
the first function (sync_ExitGracefully)
change:

Code: Select all

	LogItClose();
	unlink($newsSync_lockFile);
to

Code: Select all

       echo "about to close logfile\n";
	LogItClose();
       echo "closed logfile: about to delete .lock file\n";
	unlink($newsSync_lockFile);
       echo "deleted lock file about to call exit\n";
letmeknow.
-k



...
...
** Update Complete **
about to close logfile
closed logfile: about to delete .lock file
deleted lock file about to call exit
<br>
<b>Fatal error</b>: Maximum execution time of 30 seconds exceeded in <b>Unknown</b> on line <b>0</b><br>

it waited about 30 seconds after "deleted lock file about to call exit" and then threw the error.
qnx
Registered User
Posts: 69
Joined: Fri Mar 28, 2003 6:03 am

search db

Post by qnx »

does newssync update the phpbb search database?
does /PURGE clean that database?
Post Reply

Return to “[2.0.x] MODs in Development”