[2.0.21] Rebuild Search

All new MODs released in our MOD Database will be announced in here. All support for released MODs needs to take place in here. No new MODs will be accepted into the MOD Database for phpBB2
Forum rules
READ: phpBB.com Board-Wide Rules and Regulations

On February 1, 2009 this forum will be set to read only as part of retiring of phpBB2.

Rating:

Excellent!
58
81%
Very Good
10
14%
Good
3
4%
Fair
0
No votes
Poor
1
1%
 
Total votes: 72

cruisemates
Registered User
Posts: 77
Joined: Fri Sep 09, 2005 2:42 am
Contact:

Post by cruisemates »

Thank you for the detailed reply. I just suggested that THIS forum should use your MOD to re-index their search engine. If it is too large to do, a GOOD ALTERNATIVE for sites like mine could be to temporarily move the forum & MOD to another server & run the index. Yes, you miss a few posts, but that is small price to pay.
fuzioneer
Registered User
Posts: 28
Joined: Thu Jun 02, 2005 8:10 am

Post by fuzioneer »

this mod sounds ideal, but i am running phpnuke Platinum with integrated phpbb 2.0.11 do you think this will work ok on this configuration or has anyone already tried this on a nuked site ?
chatasos
Registered User
Posts: 748
Joined: Wed May 15, 2002 1:16 pm
Location: Paralia

Post by chatasos »

cruisemates wrote: Thank you for the detailed reply. I just suggested that THIS forum should use your MOD to re-index their search engine. If it is too large to do, a GOOD ALTERNATIVE for sites like mine could be to temporarily move the forum & MOD to another server & run the index. Yes, you miss a few posts, but that is small price to pay.


When you transfer the search tables to the new server, you can run again the mod from where it stopped in your old server, if you transfer the mod's table too to the new server. So you won't miss any posts. :wink:

Report Posts 1.2.3c (MODDB) - Report Posts 2.1.5 (ALPHA)
Rebuild Search 2.4.0 (MODDB)
MOD Version Checker 1.2.0 (MODDB)
Mega Mail System 0.9.8 (ALPHA)
Pagination Select List & Input Box (MODDB)
chatasos
Registered User
Posts: 748
Joined: Wed May 15, 2002 1:16 pm
Location: Paralia

Post by chatasos »

fuzioneer wrote: this mod sounds ideal, but i am running phpnuke Platinum with integrated phpbb 2.0.11 do you think this will work ok on this configuration or has anyone already tried this on a nuked site ?


Sorry fuzioneer, but this has only been tested in phpbb forums. But i guess if the posts and search tables are the same, there should be no problem.

Report Posts 1.2.3c (MODDB) - Report Posts 2.1.5 (ALPHA)
Rebuild Search 2.4.0 (MODDB)
MOD Version Checker 1.2.0 (MODDB)
Mega Mail System 0.9.8 (ALPHA)
Pagination Select List & Input Box (MODDB)
User avatar
ewelin
Registered User
Posts: 262
Joined: Fri May 21, 2004 2:13 pm
Location: MA, USA
Contact:

Post by ewelin »

WOW!!! great job... i really like the ACP interface. nice improvement from the previous version. Thanks for this great MOD
Prince of phpbb
Registered User
Posts: 286
Joined: Mon Jun 13, 2005 12:29 pm
Location: UK
Contact:

Post by Prince of phpbb »

Could not obtain common word list

DEBUG MODE

SQL Error : 1104 The SELECT would examine more than MAX_JOIN_SIZE rows; check your WHERE and use SET SQL_BIG_SELECTS=1 or SET SQL_MAX_JOIN_SIZE=# if the SELECT is okay

SELECT word_id FROM ns_search_wordmatch GROUP BY word_id HAVING COUNT(word_id) > 25695

Line : 309
File : functions_search.php


i got this error .. :s .. :roll:
cruisemates
Registered User
Posts: 77
Joined: Fri Sep 09, 2005 2:42 am
Contact:

Post by cruisemates »

Chatosos...

You said you were interested in screen shots when I started doing this for real. So you know, I have installed the mod both to my old server and to my current server, which I switched to last week. I started indexing my search tables on my old server (so I can disable the boards), which has 99% of the posts on it, and as soon as it is done I will transfer the tables to the new server and continue indexing.

I'm still curious to know if you think any settings would better optimize its running. I just used all the default settings.

Here are the screenshots you wanted:

http://216.119.126.62/images/searchprogress.jpg

http://216.119.126.62/images/searchprogress2.jpg

http://216.119.126.62/images/searchprogress3.jpg

Here is an update, Next morning I am still only 51% indexed and it has been 2 days, 10 hours. The original estimate was for 3 days, now it looks to be closer to 5.

Not complaining, just something to be aware of, that perhaps something is causing it to slow down. Do you think I should pause it, maybe clear up some memory?
cruisemates
Registered User
Posts: 77
Joined: Fri Sep 09, 2005 2:42 am
Contact:

Post by cruisemates »

Chatosos...

You obviously have not been monitoring this thread. I asked you before I started if there were any settings that would speed up this process. You replied to go with the defaults which I did, but I have to tell you that although everything appears to be working normally, I have now been running this indexing for 3 day, 4 hours and 15 minutes and it is still only 63.19% completed.

This is, as you can imagine, requiring a tremendous amount of patience. At this rate, it is going to take between 5 and 6 days to complete this index. Meanwhile the messages on my new server are piling up and I have no idea how long it will take to complete this index after I migrate it to my live server.

IF you have ANY suggestions on how to speed up this indexing they are GREATLY appreciated.
chatasos
Registered User
Posts: 748
Joined: Wed May 15, 2002 1:16 pm
Location: Paralia

Post by chatasos »

Prince of phpbb wrote:
Could not obtain common word list

DEBUG MODE

SQL Error : 1104 The SELECT would examine more than MAX_JOIN_SIZE rows; check your WHERE and use SET SQL_BIG_SELECTS=1 or SET SQL_MAX_JOIN_SIZE=# if the SELECT is okay

SELECT word_id FROM ns_search_wordmatch GROUP BY word_id HAVING COUNT(word_id) > 25695

Line : 309
File : functions_search.php


i got this error .. :s .. :roll:


If i am guessing right, something (MAX_JOIN_SIZE ?) must be limiting you on your sql server.
If you run the following query directly in phpmyadmin do you get the same error?

Code: Select all

SELECT word_id FROM ns_search_wordmatch GROUP BY word_id HAVING COUNT(word_id) > 25695
PS: On my server i have set MAX_JOIN_SIZE to "4294967295". Maybe you can try increasing your server's value by executing the following in phpmyadmin (if you have the sql user rights):

Code: Select all

SET max_join_size=xxxxxxxxx; 

Report Posts 1.2.3c (MODDB) - Report Posts 2.1.5 (ALPHA)
Rebuild Search 2.4.0 (MODDB)
MOD Version Checker 1.2.0 (MODDB)
Mega Mail System 0.9.8 (ALPHA)
Pagination Select List & Input Box (MODDB)
chatasos
Registered User
Posts: 748
Joined: Wed May 15, 2002 1:16 pm
Location: Paralia

Post by chatasos »

cruisemates wrote: Chatosos...

You said you were interested in screen shots when I started doing this for real. So you know, I have installed the mod both to my old server and to my current server, which I switched to last week. I started indexing my search tables on my old server (so I can disable the boards), which has 99% of the posts on it, and as soon as it is done I will transfer the tables to the new server and continue indexing.

I'm still curious to know if you think any settings would better optimize its running. I just used all the default settings.

Here are the screenshots you wanted:

http://216.119.126.62/images/searchprogress.jpg

http://216.119.126.62/images/searchprogress2.jpg

http://216.119.126.62/images/searchprogress3.jpg

Here is an update, Next morning I am still only 51% indexed and it has been 2 days, 10 hours. The original estimate was for 3 days, now it looks to be closer to 5.

Not complaining, just something to be aware of, that perhaps something is causing it to slow down. Do you think I should pause it, maybe clear up some memory?
cruisemates wrote: Chatosos...

You obviously have not been monitoring this thread. I asked you before I started if there were any settings that would speed up this process. You replied to go with the defaults which I did, but I have to tell you that although everything appears to be working normally, I have now been running this indexing for 3 day, 4 hours and 15 minutes and it is still only 63.19% completed.

This is, as you can imagine, requiring a tremendous amount of patience. At this rate, it is going to take between 5 and 6 days to complete this index. Meanwhile the messages on my new server are piling up and I have no idea how long it will take to complete this index after I migrate it to my live server.

IF you have ANY suggestions on how to speed up this indexing they are GREATLY appreciated.


Hi cruisemates,

Sorry for not replying in time, but i didn't get any email about new posts in this topic (+i'm quite busy right now with my job) so i didn't have a look at it.
As i have already told you in your pm, the processing speed depends mainly on the following things:

server hardware
server software
length of posts

On my own (hosted) server it took 2 days to process 100.000 posts. On another user's server it took 4 hours to process 40.000 posts.
But my server is collocated with another 10 servers, using the same sql db and my forum posts are generally quite big (10-20 lines). The other user's forum is on a dedicated server, with posts averaging 2-3 lines only.

If i remember correctly, you have around 750.000 posts, so a total of 6 days is around 125.000 posts per day. Depending on the 3 things i wrote above, this number can be good or bad.

Please note that mod is using phpbb's search routines, which take the most part of time while processing, so you're always limited by this.
For example, try to edit a 100 lines post in your forum (or phpbb's one) and you'll get the idea of long it takes to process such a lengthy post when thousands of posts are already in the db.


Finally, as i have already told you, you can try increasing the number of posts per cycle until you get a timeout, so you'll get fewer refreshes & fewer db updates, but that way you are rirsking of getting a timeout while processing. You can always try and see...

Report Posts 1.2.3c (MODDB) - Report Posts 2.1.5 (ALPHA)
Rebuild Search 2.4.0 (MODDB)
MOD Version Checker 1.2.0 (MODDB)
Mega Mail System 0.9.8 (ALPHA)
Pagination Select List & Input Box (MODDB)
BladeOfAnduril27
Registered User
Posts: 78
Joined: Sat Sep 27, 2003 8:54 pm
Location: Pennsylvania, USA
Contact:

Post by BladeOfAnduril27 »

Prince of phpbb wrote:
Could not obtain common word list

DEBUG MODE

SQL Error : 1104 The SELECT would examine more than MAX_JOIN_SIZE rows; check your WHERE and use SET SQL_BIG_SELECTS=1 or SET SQL_MAX_JOIN_SIZE=# if the SELECT is okay

SELECT word_id FROM ns_search_wordmatch GROUP BY word_id HAVING COUNT(word_id) > 25695

Line : 309
File : functions_search.php


i got this error .. :s .. :roll:


That should be very easy to fix. I encountered the same problem when I was rebuilding my tables. Just go into phpmyadmin, select the table, and choose "Repair" from the drop down menu. Should take care of the problem.
cruisemates
Registered User
Posts: 77
Joined: Fri Sep 09, 2005 2:42 am
Contact:

Post by cruisemates »

chatasos wrote: cruisemates, if the posts table on the new forum is copied from the old forum, then there should be no problem running my mod on the old forum and then transfering the 2 search tables to the new forum. But you'll have to transfer mod's table too, if you want to continue the rebuidling on the new forum.

Please note that if you have your new forum active, some posts may have been already indexed, so you can't transfer your old search tables unless you overwrite them.


Chatosos...

Here is my situation. I successfully used your mod to build an index of over 700,000 posts. It take about 6 days just so you know.

There is some kind of an issue that requires one to stop and restart the the mod on very long builds. The page will look like it is working, but the batch of 50 posts it is on never advances. When you stop it, you have to give it some time (5 minutes?) to finish processing whatever it is on. Then you restart it where it left off and it will start building again.

My whole indexing took about 6 days of actively running the mod, but had I figured this out earluer I think it would have only taken 4 days.

Anyway, I ran this on my old server starting from post 0 - 700,206 (the last post on that server). I already had the new boards/server running so I was not able to transfer any tables before I started indexing.

So what I want to do now is restore the search tables from the old server to the new server, and resume the search (there are about 13,000 posts on the new server not yet indexed) and resume indexing on the new server.

My questions is: WHAT EXACTLY SHOULD I DO NOW?

In terms of which tables should I transfer? I assume these four: search_rebuild, search_results, search_wordlist, search_wordmatch.

Just so you know, the sizes for each are as follows: 185, 0, 5254144, 291423232. That is about 291 MB for wordmatch. The new server has 8 MB already, but I figure I can index that in a few hours.

It likes I probably do NOT need to tramsfer search results, but please let me know. I am not a DB genius so details are appreciated.
chatasos
Registered User
Posts: 748
Joined: Wed May 15, 2002 1:16 pm
Location: Paralia

Post by chatasos »

cruisemates wrote: My questions is: WHAT EXACTLY SHOULD I DO NOW?

In terms of which tables should I transfer? I assume these four: search_rebuild, search_results, search_wordlist, search_wordmatch.

Just so you know, the sizes for each are as follows: 185, 0, 5254144, 291423232. That is about 291 MB for wordmatch. The new server has 8 MB already, but I figure I can index that in a few hours.

It likes I probably do NOT need to tramsfer search results, but please let me know. I am not a DB genius so details are appreciated.


cruisemates, you should copy the 3 search tables (search_rebuild, search_wordlist, search_wordmatch) to your new db/server and then run the mod again, choosing to continue the processing from the post you stopped (on your old server).

Report Posts 1.2.3c (MODDB) - Report Posts 2.1.5 (ALPHA)
Rebuild Search 2.4.0 (MODDB)
MOD Version Checker 1.2.0 (MODDB)
Mega Mail System 0.9.8 (ALPHA)
Pagination Select List & Input Box (MODDB)
Snapdragon
Registered User
Posts: 85
Joined: Fri Apr 04, 2003 3:45 pm
Location: Edmonton, Alberta
Contact:

Post by Snapdragon »

I tried to run this mod on our forum last night. We have 1.3 million posts and it was estimating 22 days to complete.

I find this odd because I am running a dedicated P4 3.0 box with 4 GB memory and an 8 drive RAID 10 setup for my DB server. Maybe there are some tweaks I can try? I also was using the DELETE method, I will try again tonight with the TRUNCATE method.

PS the forum is http://forums.neons.org/

We had to add a lot of stop words because the old equipment could not handle posting, never mind searching, and pruning had to be disabled because it got to the point where it could no longer prune without crashing.

Now that the forums are on my equipment, I am working hard to clean it up. I have already pruned 650,000 posts by hand, but I would really like to be able to restore full searching capability.

Since the move the forums have been amazingly fast, you can post in under a second, edit in under three, and actually load pages. Before our move it took up to a minute to post. :oops: Something to be said about doing it yourself...
chatasos
Registered User
Posts: 748
Joined: Wed May 15, 2002 1:16 pm
Location: Paralia

Post by chatasos »

Snapdragon wrote: I tried to run this mod on our forum last night. We have 1.3 million posts and it was estimating 22 days to complete.

I find this odd because I am running a dedicated P4 3.0 box with 4 GB memory and an 8 drive RAID 10 setup for my DB server. Maybe there are some tweaks I can try? I also was using the DELETE method, I will try again tonight with the TRUNCATE method.

I'm going to dissapoint you Snapdragon, but i don't think there will be any difference.
1.3m posts is quite a BIG number and i believe the mod's routine probably needs to be rewritten from scratch in order to process faster such large forums. IMHO the problem comes from the phpbb search routine which is used in this mod.

Report Posts 1.2.3c (MODDB) - Report Posts 2.1.5 (ALPHA)
Rebuild Search 2.4.0 (MODDB)
MOD Version Checker 1.2.0 (MODDB)
Mega Mail System 0.9.8 (ALPHA)
Pagination Select List & Input Box (MODDB)
Post Reply

Return to “[2.0.x] MOD Database Releases”