[BETA] Rebuild Search Mod

A place for MOD Authors to post and receive feedback on MODs still in development. No MODs within this forum should be used within a live environment! No new topics are allowed in this forum.
Forum rules
READ: phpBB.com Board-Wide Rules and Regulations

IMPORTANT: MOD Development Forum rules

On February 1, 2009 this forum will be set to read only as part of retiring of phpBB2.
Tora
Registered User
Posts: 15
Joined: Wed Dec 01, 2004 10:28 pm

Post by Tora »

Thanks for the answers, I found a working solution with a different script, setting it to handle very few posts per cycle.

Generally speaking, I find it a bit strange that I need to find mods to do that - reindexing the search should be a built-in feature in the standard version, easily customizable to work on servers even with limited ressources allowances..
Sorry, just had to get that of my chest.

Thanks for helping again!
chali
Registered User
Posts: 16
Joined: Fri Dec 10, 2004 6:01 pm

Post by chali »

Cool Tora
Could you share your solution with us? ;)

thanks!
Tora
Registered User
Posts: 15
Joined: Wed Dec 01, 2004 10:28 pm

Post by Tora »

chali wrote: Cool Tora
Could you share your solution with us? ;)

thanks!


Sure! A warning though: All these scripts for rebuilding the search index seem to produce varying results depending on the individual configurations and limitations of the server running the affected phpbb board, so there is no guarantee my solution works for anyone else. Also, I still have to reproduce my results, so maybe I was just lucky this time :wink:

I tried all relevant scripts I could find, and was getting either time-outs, memory allocation errors, internal server errors or eternal execution cycles doing nothing.

I ended up using the db_maintenance_mod, which I found on the german phpbb.de site, here (description: http://www.phpbb.de/viewtopic.php?t=60893 download: http://www.phpbb.de/moddb/download.php?id=10).
I am running this on a english language phpbb, though.

This Mod provides for a handy collection of maintenance scripts, all bundled nicely in an ACP-integrated panel. Another warning: if you have the CH-mod also, don't use the "Check post and topic tables" part of db_maintenance_mod!

In the "configuration" of this mod I set "Maximum execution time for rebuilding (in seconds)" to "60", "Fixed amount of time available for execution (in seconds)" to "30", "Maximum post size for rebuilding (in kByte)" to "100", "Minimum posts to index per step" to "1", "Use only standard PHP 3 compatible method for indexing" to "No", "Posts indexed per second when using advanced indexing method" to "1", and "Posts indexed per second when using standard indexing method" to "1".

"Rebuild search index" using the script provided by this mod would then start the re-indexing process, with very slow results, or progress. I would still get the occasional internal server error (error 500) every couple fo thousand os processed topices, but this script has a handy feature "remembering" the last successfully indexed post, and a "Restart rebuilding" option, to continue the process where it stopped, if recreation of the search index was interrupted. This way, I didn't have to start all over again when I was getting the annyoing occasional internal server errors my webhoster seems to be unable to get rid of, and could finally complete the rebuilding process within close to 4 hours, restarting the script two times.

Hope that helps, and good luck...
Ampy
Registered User
Posts: 2
Joined: Wed Dec 22, 2004 4:28 pm

Post by Ampy »

This mod workes for me!! thnx for the info!
chali
Registered User
Posts: 16
Joined: Fri Dec 10, 2004 6:01 pm

Post by chali »

Sorry, I forgot to say that this also worked great for me.

Thank you very much tora :D

EDIT: 275,000 posts here
Last edited by chali on Tue Feb 22, 2005 5:57 pm, edited 1 time in total.
-=[GRiM]=-
Registered User
Posts: 4
Joined: Thu Apr 22, 2004 4:09 am

Post by -=[GRiM]=- »

42000+ posts on 2.0.11 with MX-Portal

Worked for me after this fix:

Find:

Code: Select all

while (($row = $db->sql_fetchrow ($result)) && (onTime () == true)) {
      add_search_words('single', $row['post_id'], stripslashes($row['post_text']), stripslashes($row['post_subject']));
      $num_rows++;
   }
Replace with:

Code: Select all

while (($row = $db->sql_fetchrow ($result))/* && (onTime () == true)*/) {
      add_search_words('single', $row['post_id'], stripslashes($row['post_text']), stripslashes($row['post_subject']));
      $num_rows++;
}
I also added the decimal place "hack/mod/change".

Took over an hour to do though.
User avatar
yexusbeliever
Registered User
Posts: 586
Joined: Mon Dec 08, 2003 8:49 pm
Location: Olympus phpBB3

Post by yexusbeliever »

This looks like a great mod. Will this get release???
kurtman
Registered User
Posts: 7
Joined: Fri Mar 04, 2005 12:16 am
Location: 9 de Julio (B) - ARGENTINA
Contact:

Post by kurtman »

Hello guys !

I really couldn't do it with this mod, with or without the TIME modification. I donwloaded this: http://www.phpbb.de/moddb/download.php?id=10, and 2 minutes after I'm rebuilding the search database (8% so far, 15.000 posts)...

Thanks TORA !!

Kurt
Argentina


PS: I've used the same configuration Tora recommended.

EDITED: It took 45 minutes to rebuild 15.000+ posts... and it works PERFECT now. THANKS AGAIN.
Image
Blakers
Registered User
Posts: 4
Joined: Sat Feb 05, 2005 4:22 am
Location: NW Pennsylvania
Contact:

Post by Blakers »

303 wrote:
akslee wrote:Sod this.
I found this and it works flawlessly.

http://www.phpbbhacks.com/viewhack.php?id=434


I think it's a same hack :roll:


But it's an update and it worked great!
:Jony:
Registered User
Posts: 2
Joined: Thu Mar 24, 2005 12:37 pm

Post by :Jony: »

Tora wrote: I ended up using the db_maintenance_mod, which I found on the german phpbb.de site, here (description: http://www.phpbb.de/viewtopic.php?t=60893 download: http://www.phpbb.de/moddb/download.php?id=10).

Does somebody know if this MOD works with 2.0.13?
»»EDDY««
Registered User
Posts: 220
Joined: Mon Jun 02, 2003 10:14 am
Location: Poland/Slupsk
Contact:

Post by »»EDDY«« »

-=[GRiM]=- wrote: 42000+ posts on 2.0.11 with MX-Portal

Worked for me after this fix:

Find:

Code: Select all

while (($row = $db->sql_fetchrow ($result)) && (onTime () == true)) {
      add_search_words('single', $row['post_id'], stripslashes($row['post_text']), stripslashes($row['post_subject']));
      $num_rows++;
   }
Replace with:

Code: Select all

while (($row = $db->sql_fetchrow ($result))/* && (onTime () == true)*/) {
      add_search_words('single', $row['post_id'], stripslashes($row['post_text']), stripslashes($row['post_subject']));
      $num_rows++;
}
I also added the decimal place "hack/mod/change".

Took over an hour to do though.

Thx.
2.0.13 ok.
50k+
Make a Tiny URL in excellent domain heh.pl


cooling system & water cooling
:Jony:
Registered User
Posts: 2
Joined: Thu Mar 24, 2005 12:37 pm

Post by :Jony: »

Rebuild search index OK:
Tora's configuration
2.0.13
43k posts
100 minutes

Thank you very much
encryption
Registered User
Posts: 392
Joined: Tue Jul 08, 2003 2:07 pm
Contact:

Post by encryption »

Tora wrote: Another warning: if you have the CH-mod also, don't use the "Check post and topic tables" part of db_maintenance_mod!


assuming you are referring to Categories Heirarchy, I am using CH and ended up running this MOD before I saw this warning.... why not use this ? any reason ? I have not seen any errors on my board so far

-e-
killa101
Registered User
Posts: 265
Joined: Thu Apr 15, 2004 12:19 am

Post by killa101 »

quick question. does this mod ADD to the data already in the searh_wordlist and search_wordmatch fields or does it REPLACE the data in them?

also, what do i set for the post limit, time limit, refresh rate. running 53000+ articles (im guessing articles means posts)
alwhipp
Registered User
Posts: 17
Joined: Tue Apr 12, 2005 2:05 am
Location: BNE
Contact:

Post by alwhipp »

Works good. But I'm getting an issue with long words exceeding the max field size for a search_wordlist (postgres db). Specifically with registry entries having the slashes removed and being entered as one big word.
Whatever it is tho, I need to make sure it processes the posts and doesn't stop on invalid entries. EG; skip words >40 or 50 char.
Had a quick look in the php for the mod, but the SQL update/insert doesn't seem to be in there.
What do poeple think the easiest place to add an if(length($word)>X){} statement would be??

Vanaka
--
Al

The problem with defending the purity of the English language is that English is about as pure as a cribhouse wh0re. We don't just borrow words; on occasion, English has pursued other languages down alleyways to beat them unconscious and rifle their pockets for new vocabulary. James D. Nicoll
Post Reply

Return to “[2.0.x] MODs in Development”