[DEV] NewsSync (Usenet -> phpBB Sync)

A place for MOD Authors to post and receive feedback on MODs still in development. No MODs within this forum should be used within a live environment! No new topics are allowed in this forum.
Forum rules
READ: phpBB.com Board-Wide Rules and Regulations

IMPORTANT: MOD Development Forum rules

On February 1, 2009 this forum will be set to read only as part of retiring of phpBB2.
wineknow
Registered User
Posts: 299
Joined: Tue Mar 04, 2003 5:20 pm
Location: Calgary, Alberta, Canada
Contact:

Re: performance

Post by wineknow »

well that is odd...
try changing exit; to return;

looking at php docs there seems to be (at times) some odd things happen with command line and exit depending on your version of php

-k
qnx wrote:
wineknow wrote:after the "Update Complete" message
newssync -- closes the logfile, deletes the .lock file and then calls exit;

to track down where this bizare behaviour is occuring, edit the inc/sync_utils.php file
the first function (sync_ExitGracefully)
change:

Code: Select all

	LogItClose();
	unlink($newsSync_lockFile);
to

Code: Select all

       echo "about to close logfile\n";
	LogItClose();
       echo "closed logfile: about to delete .lock file\n";
	unlink($newsSync_lockFile);
       echo "deleted lock file about to call exit\n";
letmeknow.
-k



...
...
** Update Complete **
about to close logfile
closed logfile: about to delete .lock file
deleted lock file about to call exit
<br>
<b>Fatal error</b>: Maximum execution time of 30 seconds exceeded in <b>Unknown</b> on line <b>0</b><br>

it waited about 30 seconds after "deleted lock file about to call exit" and then threw the error.
Last edited by wineknow on Mon Jun 09, 2003 4:08 pm, edited 1 time in total.
wineknow
Registered User
Posts: 299
Joined: Tue Mar 04, 2003 5:20 pm
Location: Calgary, Alberta, Canada
Contact:

Re: search db

Post by wineknow »

qnx wrote: does newssync update the phpbb search database?
does /PURGE clean that database?


I use the functions_insert_post mod which inserts posts the same way phpbb does.

As for purge, I use phpbb's function delete_post (or whatever it is called)
I have noticed at times that the search tables get funky; I have attributed that to things going bad when newssync crashes (while testing)...

-k
qnx
Registered User
Posts: 69
Joined: Fri Mar 28, 2003 6:03 am

Re: performance

Post by qnx »

wineknow wrote: well that is odd...
try changing exit; to return;

looking at php docs there seems to be (at times) some odd things happen with command line and exit depending on your version of php

-k


tried that and still no luck. it could be that php is taking a long time doing some internal cleanup (are you using lots of arrays?).

my php is the default that comes with redhat 7.3 (ftp://updates.redhat.com/7.3/en/os/i386). I just tried building the latest php4.3.2 from php.net and the "hang bug" went away. Guess php.net has fixed this in the newer versions. I will just use this newer version to run the cron and let the apache use the default php.

thanks!
qnx
Registered User
Posts: 69
Joined: Fri Mar 28, 2003 6:03 am

Re: performance

Post by qnx »

wineknow wrote: I an not an expert in SQL (and don't really plan to learn), so if there is anyone who can tell me that I am not indexing a field, or whatever to speed things up, that would be greatly appreciated.

that said, the looking for new posts sql is ineffecient as hell, I will add it to my list of stuff to address for the next release.

I do know there is other users with newsgroups as large as 20,000 messages; and you are correct where the first sync takes a while, while future syncs do not...

-k


my "update" run still takes over 20 seconds for cases where there is NO new posts. newssync should "remember" the number of articles from the last run, and compare it. If there is nothing new, it should quit immediately. if there are 5 new articles, it should just "fetch" those and process them. Inefficient sql will delay this "process", but 0 new posts shouldn't be affected, right?
wineknow
Registered User
Posts: 299
Joined: Tue Mar 04, 2003 5:20 pm
Location: Calgary, Alberta, Canada
Contact:

Re: performance

Post by wineknow »

qnx wrote:
wineknow wrote: I an not an expert in SQL (and don't really plan to learn), so if there is anyone who can tell me that I am not indexing a field, or whatever to speed things up, that would be greatly appreciated.

that said, the looking for new posts sql is ineffecient as hell, I will add it to my list of stuff to address for the next release.

I do know there is other users with newsgroups as large as 20,000 messages; and you are correct where the first sync takes a while, while future syncs do not...

-k


my "update" run still takes over 20 seconds for cases where there is NO new posts. newssync should "remember" the number of articles from the last run, and compare it. If there is nothing new, it should quit immediately. if there are 5 new articles, it should just "fetch" those and process them. Inefficient sql will delay this "process", but 0 new posts shouldn't be affected, right?


yes things be optimized and I'll look into it.

where is it taking longer, looking for new usenet messages or looking for new messages originating within phpbb?
-k
qnx
Registered User
Posts: 69
Joined: Fri Mar 28, 2003 6:03 am

Re: performance

Post by qnx »

wineknow wrote: yes things be optimized and I'll look into it.

where is it taking longer, looking for new usenet messages or looking for new messages originating within phpbb?
-k


in my test (one forum), looking for new usenet messages took 20 seconds and looking for new phpbb message took 25 seconds. both have 0 new messages.
wineknow
Registered User
Posts: 299
Joined: Tue Mar 04, 2003 5:20 pm
Location: Calgary, Alberta, Canada
Contact:

Re: performance

Post by wineknow »

qnx wrote:
wineknow wrote:
yes things be optimized and I'll look into it.

where is it taking longer, looking for new usenet messages or looking for new messages originating within phpbb?
-k


in my test (one forum), looking for new usenet messages took 20 seconds and looking for new phpbb message took 25 seconds. both have 0 new messages.


fyi:
- I now have a single sql query for looking for new phpbb messages and it will take < 1 or 2 seconds in testing
- as for the looking for nntp messages, in short of rewriting the nntp library, there is no apparent (and easy) fix -- I will have to think about it a bit...

also, I was able to duplicate the exit routine timing out under php 4.1.2 -- it only happends with a large newsgroups. This appears to be well under the covers of php, so the workaround is to upgrade php.

-k
qnx
Registered User
Posts: 69
Joined: Fri Mar 28, 2003 6:03 am

Re: performance

Post by qnx »

wineknow wrote: fyi:
- I now have a single sql query for looking for new phpbb messages and it will take < 1 or 2 seconds in testing
- as for the looking for nntp messages, in short of rewriting the nntp library, there is no apparent (and easy) fix -- I will have to think about it a bit...

also, I was able to duplicate the exit routine timing out under php 4.1.2 -- it only happends with a large newsgroups. This appears to be well under the covers of php, so the workaround is to upgrade php.

-k


1. I can test it when you are ready.
2. "rewrite" the library? doesn't it already have function to check how many articles are on the news server?
3. yes, redhat 7.3 that I use comes with php4.1.2, which caused the "exit" hang.
qnx
Registered User
Posts: 69
Joined: Fri Mar 28, 2003 6:03 am

Post by qnx »

qnx wrote: it seems the SubjectCompare is not working in 1.0.4 and it worked fine in 1.0.3. Now we get lots of those "Unfortunately, this message is unavailable." posts and the number of "Posts" in the phpbb2 Index page doesn't match the number of articles on the news server after the initial import. I guess it is caused by those "Unfortunately, this message is unavailable." posts? If you look closely, it is available, just newssync created a new thread for that message. Can't we use the "compare subject" technique to eliminate those "Unfortunately, this message is unavailable." problems?

Just tried 1.0.3 and there is no such problem. looks like "compare subject" quit working in 1.0.4.


any updates on this issue?
wineknow
Registered User
Posts: 299
Joined: Tue Mar 04, 2003 5:20 pm
Location: Calgary, Alberta, Canada
Contact:

Post by wineknow »

I still can't duplicate the problem --- it seems to be working fine for me.
try turning split-subject off to see if you get "better" results

btw: I am heading out of town in an hour for a week...

also, I have been able to increase the speed of large newsgroups message updates -- testing soon...

-k

qnx wrote:
qnx wrote: it seems the SubjectCompare is not working in 1.0.4 and it worked fine in 1.0.3. Now we get lots of those "Unfortunately, this message is unavailable." posts and the number of "Posts" in the phpbb2 Index page doesn't match the number of articles on the news server after the initial import. I guess it is caused by those "Unfortunately, this message is unavailable." posts? If you look closely, it is available, just newssync created a new thread for that message. Can't we use the "compare subject" technique to eliminate those "Unfortunately, this message is unavailable." problems?

Just tried 1.0.3 and there is no such problem. looks like "compare subject" quit working in 1.0.4.


any updates on this issue?
qnx
Registered User
Posts: 69
Joined: Fri Mar 28, 2003 6:03 am

Post by qnx »

wineknow wrote: I still can't duplicate the problem --- it seems to be working fine for me.
try turning split-subject off to see if you get "better" results

btw: I am heading out of town in an hour for a week...

also, I have been able to increase the speed of large newsgroups message updates -- testing soon...

-k


the split-subject off didn't help.
As I PM'ed you, the new test version seems to have solved most of such problem, but still a few left unresolved. again, I tried split-subject off with no difference.

thanks for your help!
4bidden
Registered User
Posts: 8
Joined: Fri May 03, 2002 1:36 pm
Location: Samara

/data/*.* files // quote in quote // Russian

Post by 4bidden »

Hello!

At first let me tell you Thanks
for your very useful newssync system!
But I have a few little questions now...

1. What mean the information in the
/data/*.txt and /data/*.dat files?
Is that a temporary history files
and can I delete it gently?

2. That seems a problem with the "double quote",
when a "quotation" in a message
have a "quotation" inside themself...

3. Do you speak Russian (if you add a Cyrillic compatibility
in your system)?
reco
Registered User
Posts: 8
Joined: Sun May 25, 2003 9:04 pm
Contact:

Post by reco »

Hi,

I have installed PNphpBB2 as part of my postnuke intergration. I have setup newssync in a subdirectory of:

postnuke/modules/PNphpBB2/newssync/

When I try and run anything there is just a white screen and nothing happens. I see in a previous post you are considering supporting PNphpBB2 with Newssync. I at least would much appreciate this and I'm sure many would as this is a great tool

Thanks
Rudi1234
Registered User
Posts: 12
Joined: Sat Jun 21, 2003 8:30 am

Post by Rudi1234 »

Hi,

I am experimenting with this mod on our board (thanks Wineknow!) and with great results. Even though I'm synchronising with an infrequent usenet group, I wouldn't want to miss this mod anymore!

I have one question though: sometimes I want to delete one or more Usenet messages on the phpBB board. When I do, after the next synchronisation the Usenet message appears again. It seems there is no way (unless of course the Usenet message has expired) to delete a Usenet message from phpBB.

Is this true, or am I doing something wrong? Is there a workaround?

I want to present quality content on my board (www.testforum.nl) and am anxious to delete anything remotely spam-ish.

Regards,

Rudi
wineknow
Registered User
Posts: 299
Joined: Tue Mar 04, 2003 5:20 pm
Location: Calgary, Alberta, Canada
Contact:

Re: /data/*.* files // quote in quote // Russian

Post by wineknow »

4bidden wrote: 1. What mean the information in the
/data/*.txt and /data/*.dat files?
Is that a temporary history files
and can I delete it gently?

- this is a cache of the header information existing messages
- it automatically cleans itself up as the newserver deletes messages
- if you delete it - it will rebuild it at next run
4bidden wrote: 2. That seems a problem with the "double quote",
when a "quotation" in a message
have a "quotation" inside themself...


- I'll have a look
4bidden wrote: 3. Do you speak Russian (if you add a Cyrillic compatibility
in your system)?


- not a word
Post Reply

Return to “[2.0.x] MODs in Development”