[Answered] Question regarding phpbb_search_wordlist/match

This is an archive of the phpBB 2.0.x support forum. Support for phpBB2 has now ended.
Forum rules
Following phpBB2's EoL, this forum is now archived for reference purposes only.
Please see the following announcement for more information: viewtopic.php?f=14&t=1385785
Locked
User avatar
Inferno
Registered User
Posts: 42
Joined: Thu Aug 09, 2001 2:28 pm
Contact:

[Answered] Question regarding phpbb_search_wordlist/match

Post by Inferno » Thu Sep 05, 2002 3:53 am

These tables are quite large (2/3 of the db) and I would like to exclude them from my backup process. I was curious to know if I did exclude them, if, worst case scenario, I could recreate the data in those tables based on the rest of the database tables?

Or, if someone could point me in the right direction, I could potentially create the script myself. Is there a magic function somewhere that populates those tables when someone posts?

Thanks for any and all info.
Inferno
Inferno
Burn baby burn!!
www.clanmch.net/phpBB2

User avatar
Ashe
Former Team Member
Posts: 642
Joined: Sun Jul 08, 2001 11:38 am

Post by Ashe » Thu Sep 05, 2002 12:04 pm

The Unified Convertor System features a search reindexing tool that will do the job for you. As far as I know a similar function will be integrated in phpBB 2.2 admin panel.

The function you're looking for is add_search_words() in includes/functions_search.php.

User avatar
Inferno
Registered User
Posts: 42
Joined: Thu Aug 09, 2001 2:28 pm
Contact:

Post by Inferno » Thu Sep 05, 2002 3:04 pm

Thanks much!!
Inferno
Burn baby burn!!
www.clanmch.net/phpBB2

tradertt
Registered User
Posts: 180
Joined: Sat Nov 16, 2002 4:52 pm
Contact:

Post by tradertt » Tue Dec 17, 2002 1:21 am

Ashe wrote: The Unified Convertor System features a search reindexing tool that will do the job for you. As far as I know a similar function will be integrated in phpBB 2.2 admin panel.

The function you're looking for is add_search_words() in includes/functions_search.php.



How do I use this? I have downloaded the UCS but where do I find the add search word in include? What do I do with that?

asdfuae
Registered User
Posts: 46
Joined: Mon Jun 24, 2002 1:31 pm
Contact:

Post by asdfuae » Mon Jan 06, 2003 7:47 pm

Unified Convertor System doesn't rebuild none laten words like arabic correctly.

User avatar
tonmo
Registered User
Posts: 31
Joined: Mon Nov 11, 2002 9:42 pm

Post by tonmo » Sun Mar 02, 2003 6:00 pm

I downloaded the UCS file but it won't let me unpack it -- it prompts for a password for each file.

Also -- assuming I can get this password... how should I use this after unpacking?

User avatar
tonmo
Registered User
Posts: 31
Joined: Mon Nov 11, 2002 9:42 pm

Post by tonmo » Sun Mar 02, 2003 6:32 pm

More specifically -- I'm unsure what I need to do to reindex the phpbb_search_wordmatch table on my forum, which is now on a new server. Can someone outline the steps here? Thanks in advance!

-Mac-
Registered User
Posts: 65
Joined: Thu Oct 31, 2002 3:25 pm
Location: Brum, UK
Contact:

Post by -Mac- » Fri Jun 13, 2003 12:57 pm

Hi, I have just used the UCS to reconstruct the search tables for my 35'000 post forum which was converted from UBB.

I am very happy with the tool, a great job done!! But I can see why doing the reindexing is so slow - I look at my database now and the search tables are 48'000 and 538'000 rows in total - and I only got up to the 29000th post before cancelling...



I'm wondering though, this script was running for 3 hours, and it made the whole of my website lag or become unavailable.. (i don't mean my forum which was shut-down, the rest of the website hosted on the same machine)...

I really don't think my webhost would appreciate that, as it's not dedicated hosting I would imagine other people's websites were negatively affected by the script... so I'm wondering if it would be possible to add a throttle for how fast it runs, so that you can execute the script on larger forums without such a major impact.

I'm not sure if there's any way in php to say "only use xx% cpu".. but one way I thought was to run a trial first and throttle the speed based on that..

eg:
  • You load the page and select to reindex your searches.
  • It has an option saying 'test & set speed' which is recommended for large forums (>5000 posts)
  • If you select this, it runs through the first 100 posts, and measures the time taken
  • It reports back saying "executed 32 posts/sec at full load, or 0.031 posts per second"
  • You then have the manual option to specify how many posts/sec.. seconds/post.. or just %load you want to operate at.. it would be simple maths to convert between these values
  • An upshot of this is it could offer an estimated time to completion at the rate you specify
A more direct way is just to specify run at 20% load when you initially run the script, it then performs a small test (or just gagues the speed from the first 100 posts) and then adjusts its speed accordingly.

Is this a possibility, both a technical one and does the coder have time? I think it would be a worth addition, as there are hopefully more and more people coming to phpBB from other forums who need their searches fixed up!

Regards, and thanks for the mod :o)

User avatar
w3neo
Registered User
Posts: 67
Joined: Wed Oct 23, 2002 3:35 pm
Contact:

Post by w3neo » Tue Sep 02, 2003 4:46 pm

I used Rebuild Search Index on my phpBB 2.0.6 board. The installation couldnt be easier, just copy the two files to the proper directories and it shows up in the Admin Panel of your forum.

I did replace the default 100 with 25 in the admin_rebuild_search.php to slow it down a little. My new host has implemented the max questions limit so I only have 36,000(x 3 users or 108,000) per hour. Just click the rebuild index button in the Admin Panel and let it run over night.

Just for example. My database on the day I moved it was 45+ MB with over 20 MB being the two tables used for searching taking out these two tables reduced my file size by nearly half but more importantly when restoring where this max questions limit is in place was the number of lines (questions) in the mysqldump file was reduced from 1,350,000 to a paltry 250,000 lines which I was able to send to my new database (108,000 per hour max) using another script bigdump.php which you can tweak around the values to throttle it down. Just shut your forum down for the night and let it go :)

One of the many benefits of bigdump.php is that it will restore from a file you have on your server instead of coming from your local machine. You can upload a gzip file (in my case for a 45 MB database the dump was 98 MB and gzipped it was 13 MB) and use a utility like myshell.php to gunzip it on the server with the following syntax.

gunzip -d filename.gz

(NOTE: I put myshell.php and bigdum.php in my phpMyAdmin directory where they are password protected. Might not want to leave these laying around your site)

Once your forum is back on your server the Rebuild Index mod takes care of the huge wordmatch table (I ran this the following night without closing the forum).

I sure would like to see some of this built into future versions of phpBB (particularily the ability to save the database backup on the server and the the ability to throttle the restore, I think this latter will become more of an issue in the future because it is my understanding the max questions is a new feature in mysql that allows the host to set this value to disable your board if you exceed it) as it was a real pain to find and figure out these tools plus as disparate pieces I will have to remember it all next time I need to do this.

Hope this helps some of you on slow connections when dealing with these enormous (in terms of uploading and downloading) backup files.

w3
[/list]

Locked

Return to “2.0.x Support Forum”

Who is online

Users browsing this forum: No registered users and 14 guests