Disapearing Search Index using native Fulltext

Get help with installation and running phpBB 3.3.x here. Please do not post bug reports, feature requests, or extension related questions here.
rxu
Extensions Development Team
Posts: 3711
Joined: Wed Oct 25, 2006 12:46 pm
Location: Siberia, Russian Federation
Contact:

Re: Disapearing Search Index using native Fulltext

Post by rxu »

Do you have Common word threshold set to 0 in ACP/Search settings-phpBB Native Fulltext?
User avatar
rockedge
Registered User
Posts: 69
Joined: Sat Nov 30, 2019 9:10 pm

Re: Disapearing Search Index using native Fulltext

Post by rockedge »

rxu wrote: Sun Dec 05, 2021 5:48 am Do you have Common word threshold set to 0 in ACP/Search settings-phpBB Native Fulltext?
Yes it is set to zero.

Min characters indexed by search: 3
Max characters indexed by search: 14
update enabled
rxu
Extensions Development Team
Posts: 3711
Joined: Wed Oct 25, 2006 12:46 pm
Location: Siberia, Russian Federation
Contact:

Re: Disapearing Search Index using native Fulltext

Post by rxu »

rockedge wrote: Tue Dec 07, 2021 1:46 am Yes it is set to zero.
Looks like that's the cause (probably a bug). Could you try non-zero value and check if wordmatch table will stop being truncated.
User avatar
rockedge
Registered User
Posts: 69
Joined: Sat Nov 30, 2019 9:10 pm

Re: Disapearing Search Index using native Fulltext

Post by rockedge »

rxu wrote:Could you try non-zero value and check if wordmatch table will stop being truncated.
I used 70% for a tryout. So far it is working okay and the search index is remaining. I have disabled index update as well.
User avatar
warmweer
Jr. Extension Validator
Posts: 11194
Joined: Fri Jul 04, 2003 6:34 am
Location: Van Allen Bel ... gium
Contact:

Re: Disapearing Search Index using native Fulltext

Post by warmweer »

Common word threshold set to 0 would (strictly) imply that every word is considered "common" and it wouldn't be added to search table.

Code: Select all

Common word threshold:
Words which are contained in a greater percentage of all posts will be regarded as common. Common words are ignored in search queries. Set to zero to disable. Only takes effect if there are more than 100 posts. If you want words that are currently regarded as common to be reconsidered you have to recreate the index.
The set to zero to disable is ambiguous (to me). Since it immediately follows Common words are ignored in search queries. I suspect (linguistically) that it should imply that common words then wouldn't be ignored in search queries (but they shouldn't be removed as the the min and max characters criteria defines the contents of the search tables).

However, the last line

Code: Select all

If you want words that are currently regarded as common to be reconsidered you have to recreate the index.
does imply that the content of the table(s) also depends on this setting.
Spelling is freeware, which means you can use it for free.
On the other hand, it is not open source, which means you cannot change it or publish it in a modified form.


Time flies like an arrow, but fruit flies like a banana.
rxu
Extensions Development Team
Posts: 3711
Joined: Wed Oct 25, 2006 12:46 pm
Location: Siberia, Russian Federation
Contact:

Re: Disapearing Search Index using native Fulltext

Post by rxu »

Actual code acts a bit differently.
Imagine that number of posts on a board is > 100.
1st, all words will be added to search table regardless. No words are ignored in search.
2nd, if Common word threshold setting is not 0, function tidy() (triggered by cron) will select words from SEARCH_WORDLIST_TABLE where roughly word_count is greater than Common word threshold * posts number.
3rd, for all words selected on step 2 flag word_common will be set.
4th, all "common" words will be deleted from SEARCH_WORDLIST_TABLE.

Hence when you set Common word threshold to 0 all 2-4 steps will be skipped. That leads to SEARCH_WORDMATCH_TABLE gets raising with every new post. In OP case I guess IHP just truncating the table once it gets over some size limit.
This is not the case when index updating is disabled as SEARCH_WORDMATCH_TABLE is not being updated.
Post Reply

Return to “[3.3.x] Support Forum”