Recreating The Search Index

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Ideas Centre
User avatar
ChinaGal
Registered User
Posts: 90
Joined: Fri Jan 24, 2020 10:02 am

Recreating The Search Index

Post by ChinaGal »

How difficult and/or risky is it to recreate the index after setting the common word threshold to 0% under Search settings in the ACP?

Many thanks...

User avatar
warmweer
Jr. Extension Validator
Posts: 3909
Joined: Fri Jul 04, 2003 6:34 am
Location: Van Allen Bel ... gium

Re: Recreating The Search Index

Post by warmweer »

ChinaGal wrote:
Tue Feb 11, 2020 7:19 am
How difficult and/or risky is it to recreate the index after setting the common word threshold to 0% under Search settings in the ACP?

Many thanks...
The risk is that the search tables will become massive (and perhaps even too large for your hosting plan) and your board will slow down (could be serious with an active board and a lot of searching).
We should embrace problems, without which there wouldn't be any solutions.

User avatar
ChinaGal
Registered User
Posts: 90
Joined: Fri Jan 24, 2020 10:02 am

Re: Recreating The Search Index

Post by ChinaGal »

warmweer wrote:
Tue Feb 11, 2020 8:05 am
ChinaGal wrote:
Tue Feb 11, 2020 7:19 am
How difficult and/or risky is it to recreate the index after setting the common word threshold to 0% under Search settings in the ACP?

Many thanks...
The risk is that the search tables will become massive (and perhaps even too large for your hosting plan) and your board will slow down (could be serious with an active board and a lot of searching).
Thanks. So just live with it?

User avatar
warmweer
Jr. Extension Validator
Posts: 3909
Joined: Fri Jul 04, 2003 6:34 am
Location: Van Allen Bel ... gium

Re: Recreating The Search Index

Post by warmweer »

ChinaGal wrote:
Tue Feb 11, 2020 8:12 am
Thanks. So just live with it?
0% is really drastic, but you could experiment with 2% or 1% (depending how large and active your board is).
If you can use Sphinx then you shouldn't be held back.
We should embrace problems, without which there wouldn't be any solutions.

User avatar
ChinaGal
Registered User
Posts: 90
Joined: Fri Jan 24, 2020 10:02 am

Re: Recreating The Search Index

Post by ChinaGal »

warmweer wrote:
Tue Feb 11, 2020 8:32 am
ChinaGal wrote:
Tue Feb 11, 2020 8:12 am
Thanks. So just live with it?
0% is really drastic, but you could experiment with 2% or 1% (depending how large and active your board is).
If you can use Sphinx then you shouldn't be held back.
Thanks. I am not really sure I understand what the result will be of this setting. 1% or 2% would be fine. But the issue is that there are some posts on the site that are not searchable at all on some of their 4 letters keywords and will result in a message that says:
The following words in your search query were ignored because they are too common words:
You must specify at least one word to search for. Each word must consist of at least 3 characters and must not contain more than 25 characters excluding wildcards.
So I am searching with 4 character keywords for example and still getting the above message in some cases. I thought by setting it to 0% would avoid that, but then it says that you would need to recreate the search index for the change to have an effect on existing topics. So is there any solution to getting the existing topics reindexed now so that these keywords which aren't showing will show up now?

User avatar
warmweer
Jr. Extension Validator
Posts: 3909
Joined: Fri Jul 04, 2003 6:34 am
Location: Van Allen Bel ... gium

Re: Recreating The Search Index

Post by warmweer »

ChinaGal wrote:
Tue Feb 11, 2020 9:21 am
So I am searching with 4 character keywords for example and still getting the above message in some cases. I thought by setting it to 0% would avoid that, but then it says that you would need to recreate the search index for the change to have an effect on existing topics. So is there any solution to getting the existing topics reindexed now so that these keywords which aren't showing will show up now?
I had forgotten about the 3 character search requirement. There's nothing you can do about that (except perhaps or some code edits but I can't advise on that). As for the common words: that's the 3%, and I wasn't properly awake when I mentioned reducing that to 2% or 1%. That would actually increase the number of common words. Setting that to 0% disables the common word limitation. I normally use 5% and on a small board even 8%.
We should embrace problems, without which there wouldn't be any solutions.

User avatar
ChinaGal
Registered User
Posts: 90
Joined: Fri Jan 24, 2020 10:02 am

Re: Recreating The Search Index

Post by ChinaGal »

Thank you. But it says that recreating the search index will resolve this problem.

And I had did a 4 character search and got the 3 character minimum error message, which makes no sense.

I also just did a 2 word search (one of the words had 11 characters) and I got this reply:
The following words in your search query were ignored because they are too common words:
So recreating the search index I think may be the only way to wipe this problem on existing posts now that I have set it to 0%. But I am not sure how to force it to search everything.

Unfortunately the search as it is now is not an effective search if it can't search by every word and keywords containing only 4 characters. As the forum grows, assuming the problem continues, it is going to make the forum even less searchable I presume?

User avatar
warmweer
Jr. Extension Validator
Posts: 3909
Joined: Fri Jul 04, 2003 6:34 am
Location: Van Allen Bel ... gium

Re: Recreating The Search Index

Post by warmweer »

ChinaGal wrote:
Tue Feb 11, 2020 10:02 am

So recreating the search index I think may be the only way to wipe this problem on existing posts now that I have set it to 0%. But I am not sure how to force it to search everything.
Of course, the search index has to be recreated to include the new "uncommon" words.
We should embrace problems, without which there wouldn't be any solutions.

User avatar
ChinaGal
Registered User
Posts: 90
Joined: Fri Jan 24, 2020 10:02 am

Re: Recreating The Search Index

Post by ChinaGal »

warmweer wrote:
Tue Feb 11, 2020 10:11 am
ChinaGal wrote:
Tue Feb 11, 2020 10:02 am

So recreating the search index I think may be the only way to wipe this problem on existing posts now that I have set it to 0%. But I am not sure how to force it to search everything.
Of course, the search index has to be recreated to include the new "uncommon" words.
OK, thank you. So is there perhaps a safe way this can still be done somehow?

User avatar
Lumpy Burgertushie
Registered User
Posts: 67236
Joined: Mon May 02, 2005 3:11 am
Contact:

Re: Recreating The Search Index

Post by Lumpy Burgertushie »

I believe what he meant above was that setting it to zero would possilbly overload the server, not doing the reindex. I have reindexed huge boards before with no problems other than the time it takes.

the larger the posts tables the longer it takes to reindex. we are talking about hours and hours if a large board.

also, once you start it you need to leave the page and the small box open until you get a success message.
don't interupt the process or it will start over.
as usual, make a database backup before you start and if anything should go wrong you are only a few clicks away from a complete recovery.

luck,
robert
I'm baaaaaccckkkk. still doing work on donation basis. PM your needs.

Premium phpBB 3.3 Styles by PlanetStyles.net

If nobody is in the forest, does a tree really fall?

User avatar
ChinaGal
Registered User
Posts: 90
Joined: Fri Jan 24, 2020 10:02 am

Re: Recreating The Search Index

Post by ChinaGal »

Lumpy Burgertushie wrote:
Tue Feb 11, 2020 5:28 pm
I believe what he meant above was that setting it to zero would possilbly overload the server, not doing the reindex. I have reindexed huge boards before with no problems other than the time it takes.

the larger the posts tables the longer it takes to reindex. we are talking about hours and hours if a large board.

also, once you start it you need to leave the page and the small box open until you get a success message.
don't interupt the process or it will start over.
as usual, make a database backup before you start and if anything should go wrong you are only a few clicks away from a complete recovery.

luck,
robert
Understood, thank you very much. So I will put it at 2%. Otherwise, I don't think it would take more than a few hours to reindex our site. Its still very new.

Overall, it sounded risky though based on what he said in his first reply:
The risk is that the search tables will become massive (and perhaps even too large for your hosting plan) and your board will slow down (could be serious with an active board and a lot of searching).
Anyway, noted on making a backup a site backup first. We often make backups a couple of times a day anyway.

The big question is, what is the process? Is there a wiki on exactly how to do it?

I also assume the first thing to do is to take the board offline so that nobody adds any posts or tries to login while it running?

By the way, are the search tables going to become massive after the process is done and will the board slow down a lot?

Thanks again...

User avatar
Lumpy Burgertushie
Registered User
Posts: 67236
Joined: Mon May 02, 2005 3:11 am
Contact:

Re: Recreating The Search Index

Post by Lumpy Burgertushie »

the size of the search tables will depend mainly on how many words you have in all your posts combined.
also on the type of search index you choose.

I would suggest that you just leave it alone since your board is new. at some point in the future and your board gets a lot of posts and therfore words, if your search system in not working well you can think about changing things with it.

the default search index has been working just fine for most boards for years. it apparently works just fine with thousands and thousands of posts.


up to you.
by the way, the answer to your question is that it probably won't make any difference at all while your board is relativly small.
the way to do it is to just click the button that says reindex and wait until it is finished.
if you want to take the board offline while it is running it won't hurt anything. IF you don't then unless you have a very busy board it won't make any difference at all.


roberrt
I'm baaaaaccckkkk. still doing work on donation basis. PM your needs.

Premium phpBB 3.3 Styles by PlanetStyles.net

If nobody is in the forest, does a tree really fall?

User avatar
ChinaGal
Registered User
Posts: 90
Joined: Fri Jan 24, 2020 10:02 am

Re: Recreating The Search Index

Post by ChinaGal »

Thank you. Sorry, what do you mean by the "type of search index you choose"? You mean the settings I choose for phpBB Native Fulltext under Search settings?

The whole problem is there are some very key words that we are not able to search by at this point and I am hoping they would be added back in by a re-index. That was the impression I got from warmweer. And if it does that, then that would actually be a big benefit to us.

OK, last question, and I know the answer might seem obvious, but where is the reindex button? I don't think I have seen it anywhere in ACP.

And if there is no harm in running it then I might as well do it. It can run while we are asleep.

Thank you again.

UPDATE:
By the way, I have the search set to phpBB Native Fulltext

User avatar
Lumpy Burgertushie
Registered User
Posts: 67236
Joined: Mon May 02, 2005 3:11 am
Contact:

Re: Recreating The Search Index

Post by Lumpy Burgertushie »

all of those settings are in the admin panel, main page,
left menu at the bottom.
you can change the minimum /maximum characters needed, the common words settings and the type of
search index you wish to use.
then to reindex look under the maintenance tab last link in the left menu.

I believe you have to delete the current index and the create a new one. there is no"reindex" button as such.

robert
I'm baaaaaccckkkk. still doing work on donation basis. PM your needs.

Premium phpBB 3.3 Styles by PlanetStyles.net

If nobody is in the forest, does a tree really fall?

User avatar
ChinaGal
Registered User
Posts: 90
Joined: Fri Jan 24, 2020 10:02 am

Re: Recreating The Search Index

Post by ChinaGal »

Thanks. I got it now. It says Delete Index. Doesn't offer to create a new one straight away. But I assume you can create one after you delete it. I will give it a try. Thanks again.

Post Reply

Return to “phpBB Discussion”