[MODDB] Rebuild Search (2.2.1)

A place for MOD Authors to post and receive feedback on MODs still in development. No MODs within this forum should be used within a live environment! No new topics are allowed in this forum.
Forum rules
READ: phpBB.com Board-Wide Rules and Regulations

IMPORTANT: MOD Development Forum rules

On February 1, 2009 this forum will be set to read only as part of retiring of phpBB2.
Locked
chatasos
Registered User
Posts: 748
Joined: Wed May 15, 2002 1:16 pm
Location: Paralia

[MODDB] Rebuild Search (2.2.1)

Post by chatasos »

MOD Title: Rebuild Search

MOD Description: This mod allows admins to index all posts in forum rebuilding the search tables. Rebuild progress can be stopped and resumed at a later time and some time/db statistics are displayed while rebuilding.
## This mod will index every post in your forum, rebuilding the search tables.
##
## This is useful when :
##
## - You have imported posts from another forum
## - You have lost/deleted/corrupted your phpbb search tables in one way or another
## - You want to transfer your db to another server (you can skip the copying of search tables
## which are usually quite a lot of MB and create them on the new server from scratch)
## - When you have updated the "search_stopwords.txt" file (running the mod will remove these words from the search tables)
## - Users cannot find what they are searching for in your forum (although there could many other reasons for this)
##
## You can stop whenever you like and the next time you run it again, you'll have the option of
## continuing from where you left off.
##
## It may take a long time to show its progress (depending on "Posts per cycle" and "Time limit"),
## so please do not move from its progress page until it is complete, unless of course you want to interrupt it.




MOD Version: 2.2.1 (RC)

MOD Download: rebuild_search_2.2.1.zip


This mod is based on GUI's original MOD (1.0.0), but during its development it almost got -completely- rewritten. Many thanks to GUI for letting me continue on his mod.

Version History:

Code: Select all

##	2005-09-15 - Version 2.2.1
##	- Moved some html into a seperate template
##	- Various small fixes
##
##   2005-09-03 - Version 2.2.0b
##	- Changed the total posts calculation function
##
##   2005-09-03 - Version 2.2.0a
##	- Fixed a mysql incompatibility
##
##   2005-09-02 - Version 2.2.0
##	- Changed mod's db table (final)
##	- Changed color percent calculation
##	- Fixed db size calculations (sql problem)
##	- Cleaned up code a little
##
##   2005-08-31 - Version 2.1.9 (beta)
##	- Added animated progress bar (courtesy of phpBB3)
##	- Fixed bug of deleting posts while processing
##
##   2005-08-31 - Version 2.1.8 (beta)
##	- Changed mod's db table (again!)
##	- Changed db size calculations (much better now)
##	- Cleaned up code a little
##
##   2005-08-30 - Version 2.1.7 (beta)
##	- Added check if board is already disabled
##	- Updated the instructions (about phpBB search operation)
##
##   2005-08-30 - Version 2.1.6 (beta)
##	- Changed mod's db table (again)
##	- Cleaned up code a lot
##	- Redesigned the progress screen (much better now)
##	- Added session/total information
##	- Added option for DELETE/TRUNCATE when clearing the search tables
##	- Added option for disabling the board while processing
##	- Changed time limit calculations (again)
##	- Optimized some functions
##	- Updated the instructions
##
##   2005-08-28 - Version 2.1.5 (beta)
##	- Changed mod's db table (hopefully faster now)
##	- Optimized some functions
##	- Added phpBB search operation details in instructions
##
##   2005-08-27 - Version 2.1.4 (beta)
##	- Rewritten time limit function in order to avoid timeouts (much better now)
##	- Updated the instructions
##
##   2005-08-27 - Version 2.1.3a (beta)
##	- Added check for webserver timeout in time limit
##	- Fixed various IE incompatibilities
##
##   2005-08-27 - Version 2.1.3 (beta)
##	- Fixed estimated db size calculation
##	- Enabled the time limit option
##	- Added check for safe mode in time limit
##	- Optimized some functions
##	- Updated the instructions a little
##	- Various minor fixes
##
##   2005-08-26 - Version 2.1.2 (beta)
##	- Changed process aborting code
##	- Fixed submit handling code, added input validation
##	- Added $lang vars where missing
##
##   2005-08-24 - Version 2.1.1 (beta)
##	- Changed the rebuild_status code
##	- Fixed a sql query to be compatible with mySQL < 4.0.0 (thanks smokedog)
##	- Updated the instructions a little
##
##   2005-08-22 - Version 2.1.0 (beta)
##	- Completely new version by chatasos, almost written from scratch.
##	- Many new features added (see author notes for details)
##
##   2003-10-03 - Version 1.0.0
##     - Initial Release by GUI
The mod takes only 2-3 minutes to install, but you'd better read the following instructions if you don't know what to do after installing.

--------------------------------------------------------------------------------
## Author Notes:
##
## - The original author (until 1.0.0) was GUI < spam@nickm.org > (Nick Muerdter).
## After that version, he gave me (chatasos) the permission to continue with his mod.
## This is almost a completely new mod based on GUI's original idea.
##
## - If you want to upgrade from GUI's version to this one, just follow the instructions for installing.
## Upgrading or installing from scratch requires exactly the same steps.
##
## - When upgrading from versions 2.1.0 - 2.1.7 to 2.1.8 or later, you have to run the db command "DROP table `phpbb_search_rebuild`"
## before proceeding with the install, because the mod's table has changed its structure.
##
## - You need to have javascript enabled in your browser in order to have the script work flawlessly.
## It may seem to work fine with javascript disabled, but you'll lose a lot of its functionality.
##
## - When running the mod it's better to have your forum disabled, because db usage is getting very high.
## Mod under normal circumstances doesn't have any problems when some posts are deleted/inserted while
## processing is on, since it automatically recalculates its parameters. But to be on the safe side,
## try to have your board disabled while processing.
##
## - There might be some problems when calculating the db sizes on non-MySQL databases,
## so please if you find any, inform the relevant topic on the phpBB site.
##
##
## phpBB search operation (not verified)
## ----------------------
## The search function of phpBB uses 3 tables:
## - search_wordlist (this contains a unique entry for each word that has been posted in the forum)
## - search_wordmatch (this contains pairs of data of which posts include which words from the search_wordlist)
## - search_results (this contains the last searches the users executed)
##
## phpBB marks heavily used words by changing the "word_common" field in the search_wordlist from 0 to 1 and then
## deleting them from the search_wordmatch table.
## Heavily used words are those who appear in the search_wordmatch table more than (4/10 * total_posts) times.
## phpBB doesn't check for common words before adding them into the tables. It justs adds them
## and then deletes them if they are found to be common.
##
## Also phpBB uses a blacklist of words (a text file containing one word per line) that you don't want to be indexed.
## If you are using the default english language, this file is "language/lang_english/search_stopwords.txt".
##
## If you want to find your most commonly used words, so you can put some of them into the blacklist,
## you can run the following db query in phpmyadmin:
##
## SELECT swl.word_id, swl.word_text, COUNT(swm.word_id) as total
## FROM `phpbb_search_wordlist` swl, `phpbb_search_wordmatch` swm
## WHERE swl.word_id = swm.word_id
## GROUP BY swm.word_id
## ORDER BY total
## DESC LIMIT 0,50
##
##
## Main page
## ---------
## You can find the mod under the Admin CP -> General Admin -> Rebuild Search.
## When you run the mod, you can change the following parameters:
##
## 1) Starting post_id
## You can choose from which post to start processing, according to some conditions:
##
## a) Start from post_id 0, which means from the 1st post of your forum
## b) Start from the post_id which you stopped the last time you ran the mod
##
## - If you haven't run the mod at all, then you can only start from the beginning.
## - If you have run the mod but there have been new posts after that, then you can start from the beginning
## (so process all posts), or continue from the post you last processed (so process only the new posts).
## - If you have run the mod but aborted/stopped during its operation, then you can start from the beginning
## (so process all posts), or continue from the post you last aborted/stopped (so process only the remaining posts).
##
## According to your choice on this option, some other values (No of posts, Posts per cycle) are changed automatically to help you.
##
## If you are getting strange post numbers while processing, you can try to start from the beginning hoping
## they'll be corrected.
##
## The post_id doesn't refer to the a/a number of a post while counting posts.
## When you have 5 sequencial posts, their numbers are 1,2,3,4,5 but their post_ids can be 2,7,10,23,56.
##
## Note : At this time you cannot change this value by hand. This is something that might change in the future,
## although i can't find any real reason for implementing such a feature.
##
## 2) Clear search tables
## When you start from the beginning (post_id 0) you have the option of clearing the forum search tables.
## In all the other cases, clearing of search tables is disabled and should not be used.
##
## You have 3 options here :
##
## NO: do not clear the search tables
## DELETE: all records from the 3 phpbb search tables are deleted (slower, safer, working on all databases)
## TRUNCATE: all 3 phpbb search tables are truncated (faster, riskier, possibly working on all databases)
##
## You should choose TRUNCATE if you also want to zero the auto_increment values of the search tables.
##
## When clearing is enabled, phpbb search tables are cleared (initialized) before starting the processing.
## Regardless of clearing is enabled or disabled, post processing always includes removal and insertion of posts.
## This is so, because if you try to insert a post without removing it first, you might come up with "double" records
## in phpbb search tables because at the same time someone else in the forum could be editing that post.
##
## 3) No of posts
## You can choose how many posts to process totally. This value is automatically filled with the total
## posts in your forum (when you haven't processed any posts, or you have processed all posts),
## or with the remaining posts (when you have stopped the processing at a previous time).
##
## Of course you can put your own value here.
##
## 4) Posts per cycle
## You can choose how many posts to process per cycle. The word "cycle" refers to the amount of posts
## belonging to the same "group" which is proccesed one at a time (it's like a batch of posts).
## So, if you choose to process 200 posts totally, at 20 posts per cycle, you'll spend 200/20=10 cycles
## in order to complete the whole processing.
##
## If you are getting timeouts while processing you should try lowering this value (see also the Time limit option).
## Also small numbers (20-50) will help you watch much easier the progress.
##
## Normally the mod calculates processing times per post and adjusts (decreases) this value automatically,
## so it shouldn't timeout whatever value you use here. But always there can be exceptions...
##
## Of course if you think you have a fast server/db setup you can always risk increasing it.
##
## 6) Time limit
## You can choose the maximum time in seconds the mod is allowed to run before it moves to the next processing cycle.
## The mod is using as default the value of 300 secs, which is usually found on many webservers.
##
## If you are getting timeouts while processing, you should try decreasing it (see also the Posts per cycle option).
##
## If you're using Mozilla, mod will be probably able to get the webserver's timeout and inform you about this,
## so you'd better keep your value below this. IE doesn't support this feature.
##
## Also older IE are known to have problems with large waiting times, so you'd better try an alternative browser like Mozilla.
##
## If your php is running in safe mode, mod will inform you of php's timeout, so you'd better keep your
## value below this too.
##
## 5) Refresh rate
## You can choose how much time to wait after a cycle has finished and until the next one begins.
## This is called the refresh time and it's time that actually nothing happens.
## You can always press the "Next" button if you don't want to wait until the timer expires,
## but you cannot press it while processing is on (the button displays "Processing..." during that time),
## because you are in the risk of bypassing some posts while processing and messing with the db.
##
## Note : This feature might seem quite helpful when "debugging" mod's usage: you can put a low number
## of posts per cycle and a high refresh rate, so you can watch the progress screen as it changes
## and take screenshots easily.
##
## 6) Disable board
## You can disable your board while processing so online users don't mess with the db. Normally there should be
## no problem if you leave it enabled, since the mod is acting like "editing" posts when processing them
## and as you know whoever gets there last is the one who wins. The main reason why you'd want to disable it,
## is if you have many users online and the db usage gets high, so prevent the forum from becoming sluggish.
##
## If your board is already disabled, you can't change this option.
##
##
## Progress Screen
## -------------------
## When you press the "Rebuild Search" button, the button label changes to "Processing..." and becomes disabled
## until the first group of posts has been processed (or time limit has expired).
## After that, a progress screen appears which displays various information about the posts already
## processed and the ones that follow, like processing time details and database size details.
## Also the percentage of completed posts (session/total) is displayed in both numerical and graphical format.
##
## When you see the message "Timer expired at x secs", it means that the mod couldn't process all the posts you told it to
## and moved to the next cycle before the timer you have defined expires. Everything is continuing to work
## fine, so there is no real need to worry, but you may want to lower the "Posts per cycle" value in order to
## avoid such timeouts.
##
## In order for the mod to move to the next processing cycle, one of the following conditions must be met:
##
## a) All "Posts per cycle" have been processed
## b) "Time limit" has expired
##
## As noted on the bottom of the screen, all estimated values are calculated approximately
## based on current percent of completion. I know this might not be the best way to do it,
## but it works quite well as the percent increases and it provides some interesting infos.
##
## Please note that the db sizes display wrong values when you don't clear the search tables.
##
##
## Operation aborting
## ------------------
## The "Cancel" button at the bottom allows you to interrupt the processing of posts at your will.
##
## If you press it while processing is on (left button says "Processing..." during that time), the group of posts
## that was being processed at that time will continue to be processed, although you may think that the script has stopped.
## This is default php/apache behaviour and i don't think i can do much about it.
##
## If you press it while the refresh timer is shown (left button says "Next" during that time), the processing of the last
## group of posts will have finished, so the script has finished too.
##
## If you have pressed it during processing, then please take note of the following:
##
## When you return to the main page, if you see a message "please wait..." at the bottom, this means that probably the
## processing of the last cycle hasn't finished yet and you'll have to wait a little more. So keep refreshing this page until
## the message disappears (you'll see that the starting post_id has changed when it finishes).
## Of course you can continue if you like without waiting, but keep in mind that you'll be beginning a new processing session
## while the previous hasn't finished yet, whatever that means for your database.
##
##
## NOTE : This mod indexes your posts (by rebuilding the search tables) exactly the same way phpbb does on each post
## when it's written. So if afterwards you cannot find some words of a specific post that has been indexed,
## try to edit that post in phpbb and check again if you can find them. If you cannot find them,
## then search for these words isn't "working" on your forum (there could be various reasons for this).


--------------------------------------------------------------------------------

Below you'll find some screenshots, where you can see the mod's operation.

While accepting initial input from admin:
Image

While processing a number of posts:
Image

After processing has finished (times are 00 because there aren't many posts):
Image

While accepting input from admin in order to continue from where he last stopped:
Image
--------------------------------------------------------------------------------

NOTE : This mod indexes your posts (by rebuilding the search tables) exactly the same way phpbb does on each post when it's written. So if afterwards you cannot find some words of a specific post that has been indexed, try to edit that post in phpbb and check again if you can find them. If you cannot find them, then search for these words isn't "working" on your forum (there could be various reasons for this).

PS: Sorry for the long writing, but i wanted to make mod's operation as clear as possible. :lol:
Last edited by chatasos on Tue Oct 04, 2005 3:20 pm, edited 34 times in total.

Report Posts 1.2.3c (MODDB) - Report Posts 2.1.5 (ALPHA)
Rebuild Search 2.4.0 (MODDB)
MOD Version Checker 1.2.0 (MODDB)
Mega Mail System 0.9.8 (ALPHA)
Pagination Select List & Input Box (MODDB)
smokedog
Registered User
Posts: 2
Joined: Sun Aug 21, 2005 9:35 am

I have tested

Post by smokedog »

I just wanted to say I have just tested this mod on my 2.0.17 install, and there were a few things that I wanted to point out, as this is a beta:

1. The SQL in the install.txt file has a trailing curly bracket } in it on the last line. This was causing a syntax error, and worked fine and created the new table when I changed it to a normal bracket ).

2. I have a different style than subSilver installed, and when I tried to use the mod after uploading the files, I get an error message advising that 'rebuild_search.tpl not found'. I therefore uploaded the files from \templates\subSilver\admin to \admin and this then worked fine. I think it was looking in the root admin folder of the forum anyway, rather than in the templates folder.

3. Once I had done this, I ran the search through to the end, which seemed to be working fine until I got an error:

Code: Select all

Could not update rebuild search data

DEBUG MODE

SQL Error : 1064 You have an error in your SQL syntax near 'ORDER BY rebuild_id DESC LIMIT 1' at line 4

UPDATE phpbb_search_rebuild SET rebuild_status = 1 WHERE end_post_id = 11000 ORDER BY rebuild_id DESC LIMIT 1

Line : 440
File : admin_rebuild_search.php
I only have 10822 posts in my forum according to phpbb and your mod, but the mod is trying to start at post 11001 when I re-run it.

I left the default settings in the parameters (i.e. 50 posts per cycle) and the 'No of posts' value says 10822, but the 'starting post_id' value says '11001 (continue from last stopped)' so I think it got through posts 10800-10822 (i.e. the last batch of supposedly 50 posts) and is trying to start from the next batch of 50 which does not exist... but I'll leave that up to you to decide what is happening :)
At this time you cannot change this value by hand. This is something that might change in the future, although i can't find any real reason for implementing such a feature.

This might be a case when that is useful :)

I hope this helps anyway, and I think its going to be a very useful mod.
chatasos
Registered User
Posts: 748
Joined: Wed May 15, 2002 1:16 pm
Location: Paralia

Re: I have tested

Post by chatasos »

smokedog wrote: I just wanted to say I have just tested this mod on my 2.0.17 install, and there were a few things that I wanted to point out, as this is a beta:


Thanks for testing it smokedog....
smokedog wrote: 1. The SQL in the install.txt file has a trailing curly bracket } in it on the last line. This was causing a syntax error, and worked fine and created the new table when I changed it to a normal bracket ).

Yep, that must have slipped away. It has been corrected now.
smokedog wrote: 2. I have a different style than subSilver installed, and when I tried to use the mod after uploading the files, I get an error message advising that 'rebuild_search.tpl not found'. I therefore uploaded the files from \templates\subSilver\admin to \admin and this then worked fine. I think it was looking in the root admin folder of the forum anyway, rather than in the templates folder.


In the instrutions, when it says:

Code: Select all

copy templates/subSilver/admin/rebuild_search.tpl to templates/subSilver/admin/rebuild_search.tpl
copy templates/subSilver/admin/rebuild_search_progress.tpl to templates/subSilver/admin/rebuild_search_progress.tpl
if you have other templates installed too, you must do the same steps for them too by replacing "subSilver" in the destination filename with your template names, ex:

Code: Select all

copy templates/subSilver/admin/rebuild_search.tpl to templates/mytemplatename/admin/rebuild_search.tpl
copy templates/subSilver/admin/rebuild_search_progress.tpl to templates/mytemplatename/admin/rebuild_search_progress.tpl
smokedog wrote: 3. Once I had done this, I ran the search through to the end, which seemed to be working fine until I got an error:

Code: Select all

Could not update rebuild search data

DEBUG MODE

SQL Error : 1064 You have an error in your SQL syntax near 'ORDER BY rebuild_id DESC LIMIT 1' at line 4

UPDATE phpbb_search_rebuild SET rebuild_status = 1 WHERE end_post_id = 11000 ORDER BY rebuild_id DESC LIMIT 1

Line : 440
File : admin_rebuild_search.php
I only have 10822 posts in my forum according to phpbb and your mod, but the mod is trying to start at post 11001 when I re-run it.

I left the default settings in the parameters (i.e. 50 posts per cycle) and the 'No of posts' value says 10822, but the 'starting post_id' value says '11001 (continue from last stopped)' so I think it got through posts 10800-10822 (i.e. the last batch of supposedly 50 posts) and is trying to start from the next batch of 50 which does not exist... but I'll leave that up to you to decide what is happening :)

I guess you must by using a mySQL version lower than 4.0.0. I changed the sql query, so now it should work with all versions of mySQL.
Also note that the "starting post_id" refers to the id of the post and not to the number of post while counting :wink: For example when you have 10 posts, their ids can be 1,5,8,10,15,18,20,49,78,90. but their numbers are 1,2,3,4,5,6,7,8,9.10.

In your case, all the posts were indexed correctly into the the search tables (11000 is the id of your 10822nd post), but since the rebuilding stopped just after the last batch (THIS is the word i was searching) of posts, the mod thinks that there are more posts to process.
smokedog wrote:
At this time you cannot change this value by hand. This is something that might change in the future, although i can't find any real reason for implementing such a feature.

This might be a case when that is useful :)

This wouldn't really help you in your case, because regardless of from which post you would start, processing would always stop where it stopped now. :wink:


I'll upload a new version later tonight but you'll have to run the rebuilding from the beginning again :( (after installing the new version) in order to update the mod's db table with correct values.

Report Posts 1.2.3c (MODDB) - Report Posts 2.1.5 (ALPHA)
Rebuild Search 2.4.0 (MODDB)
MOD Version Checker 1.2.0 (MODDB)
Mega Mail System 0.9.8 (ALPHA)
Pagination Select List & Input Box (MODDB)
chatasos
Registered User
Posts: 748
Joined: Wed May 15, 2002 1:16 pm
Location: Paralia

Post by chatasos »

Version 2.1.1 is out:

Code: Select all

##   2005-08-24 - Version 2.1.1 (beta)
##	- Changed the rebuild_status code
##	- Fixed a sql query to be compatible with mySQL < 4.0.0 (thanks smokedog)
##	- Updated the instructions a little
See 1st post for download link and details.

Report Posts 1.2.3c (MODDB) - Report Posts 2.1.5 (ALPHA)
Rebuild Search 2.4.0 (MODDB)
MOD Version Checker 1.2.0 (MODDB)
Mega Mail System 0.9.8 (ALPHA)
Pagination Select List & Input Box (MODDB)
User avatar
3Di
Former Team Member
Posts: 16091
Joined: Mon Apr 04, 2005 11:09 pm
Location: Milano 🇮🇹 Frankfurt 🇩🇪
Name: Marco
Contact:

Post by 3Di »

looks like a must have, thanks.
/me downloading it
🆓 Free support for our extensions also provided here: phpBB Studio
🚀 Looking for a specific feature or alternative option? We will rock you!
Please PM me only to request paid works. Thx. Want to compensate me for my interest? Donate
My development's activity º PhpStorm's proud user º Extensions, Scripts, MOD porting, Update/Upgrades
smokedog
Registered User
Posts: 2
Joined: Sun Aug 21, 2005 9:35 am

Re: I have tested

Post by smokedog »

chatasos wrote: In the instrutions, when it says:

Code: Select all

copy templates/subSilver/admin/rebuild_search.tpl to templates/subSilver/admin/rebuild_search.tpl
copy templates/subSilver/admin/rebuild_search_progress.tpl to templates/subSilver/admin/rebuild_search_progress.tpl
if you have other templates installed too, you must do the same steps for them too by replacing "subSilver" in the destination filename with your template names, ex:

Code: Select all

copy templates/subSilver/admin/rebuild_search.tpl to templates/mytemplatename/admin/rebuild_search.tpl
copy templates/subSilver/admin/rebuild_search_progress.tpl to templates/mytemplatename/admin/rebuild_search_progress.tpl


Yes, you're right - I guessed that at first, and copied them in there, but I just must have thought to copy them in the root admin folder too, and thought that was where they needed to be! :D - sorry about that, the files existed in both admin folders at the same time, so sorry for pointing out a non-existent bug :roll:

chatasos wrote: I guess you must by using a mySQL version lower than 4.0.0. I changed the sql query, so now it should work with all versions of mySQL.


yes, I have 3.23.58 - I'll have to see if I can get my host to update that...
chatasos wrote: Also note that the "starting post_id" refers to the id of the post and not to the number of post while counting :wink: For example when you have 10 posts, their ids can be 1,5,8,10,15,18,20,49,78,90. but their numbers are 1,2,3,4,5,6,7,8,9.10.

In your case, all the posts were indexed correctly into the the search tables (11000 is the id of your 10822nd post), but since the rebuilding stopped just after the last batch (THIS is the word i was searching) of posts, the mod thinks that there are more posts to process.


Aha, didn't realise that, but thanks for explaining, makes more sense now :)
chatasos wrote: This wouldn't really help you in your case, because regardless of from which post you would start, processing would always stop where it stopped now. :wink:


Ah right, due to the mySQL version issue, OK, thanks.
chatasos wrote: I'll upload a new version later tonight but you'll have to run the rebuilding from the beginning again :( (after installing the new version) in order to update the mod's db table with correct values.


That's no problem, and thanks for taking in my comments on your hard work, I hope that it helped, and I'll check out the new version :)

Keep up the good work, this is proving to be a very useful mod :mrgreen:
chatasos
Registered User
Posts: 748
Joined: Wed May 15, 2002 1:16 pm
Location: Paralia

Post by chatasos »

Version 2.1.2 is out:

Code: Select all

##   2005-08-26 - Version 2.1.2 (beta)
##	- Changed process aborting code
##	- Fixed submit handling code, added input validation
##	- Added $lang vars where missing
##
See 1st post for download link and details.

Report Posts 1.2.3c (MODDB) - Report Posts 2.1.5 (ALPHA)
Rebuild Search 2.4.0 (MODDB)
MOD Version Checker 1.2.0 (MODDB)
Mega Mail System 0.9.8 (ALPHA)
Pagination Select List & Input Box (MODDB)
Joe Belmaati
Registered User
Posts: 2110
Joined: Sun Sep 28, 2003 7:35 pm
Location: Denmark

Post by Joe Belmaati »

I just did a rebuild on a test board with less than 100 posts with no problems. Tonight I will try it on my main board with 40000+ posts. Will let you know. BTW, I really like the new interface :D
Image
chatasos
Registered User
Posts: 748
Joined: Wed May 15, 2002 1:16 pm
Location: Paralia

Post by chatasos »

Joe Belmaati wrote: I just did a rebuild on a test board with less than 100 posts with no problems. Tonight I will try it on my main board with 40000+ posts. Will let you know. BTW, I really like the new interface :D


Maybe you could post some screenshots while processing, so we can see some time/db values? :wink:
My forum (~100.000 posts) takes about 2 days to rebuild!

Report Posts 1.2.3c (MODDB) - Report Posts 2.1.5 (ALPHA)
Rebuild Search 2.4.0 (MODDB)
MOD Version Checker 1.2.0 (MODDB)
Mega Mail System 0.9.8 (ALPHA)
Pagination Select List & Input Box (MODDB)
Joe Belmaati
Registered User
Posts: 2110
Joined: Sun Sep 28, 2003 7:35 pm
Location: Denmark

Post by Joe Belmaati »

With the old MOD rebuliding my forum took about 40 minutes.
Image
chatasos
Registered User
Posts: 748
Joined: Wed May 15, 2002 1:16 pm
Location: Paralia

Post by chatasos »

I'm planning of adding an option where you can choose to enable/disable search processing from phpbb.
That way common forum actions (insert/edit/delete post) will become much quicker, especially on very large boards.
  1. If you choose to disable phpbb search processing, you'll have 2 more options when running the mod:
    1. Use the search rebuild function for all posts or only the new ones, so there won't be any actual change from its current operation. If you choose to process only the new posts, edited/deleted ones won't be indexed until you run the mod again from the beginning.
    2. Use the search rebuild function for only the new posts (as you can do now) and another function that will process only the edited/deleted posts. This means that we have to keep records of these 2 actions in order to be able to execute them at a later time.
    Advantages-Disadvantages
    • On both options, phpbb search tables won't be updated in realtime. You'll have to do it manually at a later time (maybe cron would help here?)
    • On both options, you have to run (& complete) at least one time the rebuild function for all posts.
    • a is the fastest one for users
    • b is slower than 1a (but faster than normal phpbb) for users
    • a takes the longest time for admins
    • b takes shorter time than 1a for admins
  2. If you choose to enable phpbb search processing (as it is now), there won't be any change from its current operation.

Report Posts 1.2.3c (MODDB) - Report Posts 2.1.5 (ALPHA)
Rebuild Search 2.4.0 (MODDB)
MOD Version Checker 1.2.0 (MODDB)
Mega Mail System 0.9.8 (ALPHA)
Pagination Select List & Input Box (MODDB)
Joe Belmaati
Registered User
Posts: 2110
Joined: Sun Sep 28, 2003 7:35 pm
Location: Denmark

Post by Joe Belmaati »

OK, here are some stats from my search table rebuild - just completed.

Code: Select all

10%     + 8 minutes     8 minutes
20%     + 16 minutes    8 minutes
30%     + 24 minutes    8 minutes
40%     + 33 minutes    9 minutes
50%     + 41 minutes    8 minutes
60%     + 49 minutes    8 minutes
70%     + 58 minutes    9 minutes
80%     + 66 minutes    8 minutes
90%     + 74 minutes    8 minutes
100%    + 82 minutes    8 minutes
Here are some screen shots for the specifics. Click on each image.

Just started
Image
10%
Image
20%
Image
30%
Image
40%
Image
50%
Image
60%
Image
70%
Image
80%
Image
90%
Image
100%
Image

Under normal circumstances I would have probably set my forum to rebuild late at night when the server isn't working so hard, so I think these stats more or less mirror those of the old MOD. I did goof up big time, though, as I forgot to record the search table sizes before starting the script, so I have no way of really knowing whether the tables grew larger.

One post came in during the rebuild. This did not constitute any problem - the post was properly search indexed.

Sincerely,
Joe Belmaati
Copenhagen Denmark
Image
chatasos
Registered User
Posts: 748
Joined: Wed May 15, 2002 1:16 pm
Location: Paralia

Post by chatasos »

WOWWWWWWWW!!!!!!!!!! I can't believe it...

You are doing 50 posts every 3 secs, while i'm doing 50 posts every 80 secs! My server sucks :cry:
40.000 posts processed under 40 mins is a great time!!!

If you have used 1" as refresh time, then your total time would be ~55 mins.
Now you have spend half your time waiting for refreshes :wink:

Also, since you seem to have a fast server (or too small posts 8) ), you could try using bigger values in post per cycle (ex. 500), which would mean less cycles, so less total time.
Last edited by chatasos on Fri Aug 26, 2005 7:12 pm, edited 1 time in total.

Report Posts 1.2.3c (MODDB) - Report Posts 2.1.5 (ALPHA)
Rebuild Search 2.4.0 (MODDB)
MOD Version Checker 1.2.0 (MODDB)
Mega Mail System 0.9.8 (ALPHA)
Pagination Select List & Input Box (MODDB)
Joe Belmaati
Registered User
Posts: 2110
Joined: Sun Sep 28, 2003 7:35 pm
Location: Denmark

Post by Joe Belmaati »

K, I will try a rebuild using those values and let you know.
Image
Joe Belmaati
Registered User
Posts: 2110
Joined: Sun Sep 28, 2003 7:35 pm
Location: Denmark

Post by Joe Belmaati »

Code: Select all

10%		 +4 minutes		4 minutes
20%		 +7 minutes		3 minutes
30%		+11 minutes		4 minutes
40%		+15 minutes		4 minutes
50%		+21 minutes		6 minutes
60%		+24 minutes		3 minutes
70%		+28 minutes		4 minutes
80%		+33 minutes		5 minutes
90%		+37 minutes		4 minutes
100%	  +42 minutes		5 minutes
Image

On my board most of the posts are 3 or so lines. There's the occasional one line and occassionally a long post (30+ lines):
Image
Locked

Return to “[2.0.x] MODs in Development”