[Beta] Sphinx search for phpBB 1.0.beta2

A place for MOD Authors to post and receive feedback on MODs still in development. No MODs within this forum should be used within a live environment!
Suggested Hosts
Locked
Arantor
Registered User
Posts: 1
Joined: Sat May 30, 2009 8:21 am
Location: http://sphinxsearch.com/

Re: [Beta] Sphinx search for phpBB 1.0.beta2

Post by Arantor » Sat May 30, 2009 8:31 am

To clarify a few things:

searchd should run in the background, like Apache and MySQL do - and process searches once they are received. Thus it should run once on a system (generally, though you can run multiple instances for certain reasons)

The plugin at first glance does seem to have some serious issues - pushing for a reindex far, far too regularly (I have heard that it triggers off every new post!) and stopping searchd repeatedly. Note that instances where searchd appears to be crashing are not - searchd is being forced to die with killall -9. See shutdown_searchd() in /sphinx_plugin/includes/search/fulltext_sphinx.php (called twice by create_index(), for example!)

This repeated process killing is the cause of 'no results' (since I've seen cases where it happens every 30 seconds!)


I won't tend to come on here too often and provide Sphinx support - I do however check out Sphinx's own support forum many times a day - I'm Arantor there too.

I will also add that we have discussed this plugin over there, and I was not entirely complimentary about it, and have actually said I'll build my own. I will be targetting phpBB3 and provide support for via the Sphinx forum.

User avatar
naderman
Consultant
Consultant
Posts: 3735
Joined: Fri Aug 01, 2003 10:06 pm
Location: Berlin, Germany
Name: Nils Adermann
Contact:

Re: [Beta] Sphinx search for phpBB 1.0.beta2

Post by naderman » Sun May 31, 2009 7:59 pm

wout000 wrote:error: The file /usr/local/bin/indexer is not executable for the webserver.

I set the config and data folder in the settings (created 2 folders that are writable) and told phpbb to use sphinx and then got the above error
You will have to adjust permissions of the indexer file so the executable bit is set for the webserver.

Letus2001 wrote:latest version of sphinx under SUSE enterprise linux doesnt have "indexer" and "searchd" binaries, but
"sphinx-indexer" and "sphinx-searchd", so this will be problem for this mod as well. Go to your /usr/bin directory (or where you have your sphinx binaries) and create symbolic links

ln -s sphinx-indexer indexer
ln -s sphinx-searchd searchd

to create the shortcuts named "indexer" and "searchd".
That's indeed a problem, I will have to add an option to configure the filename of the executables.
Arantor wrote:The plugin at first glance does seem to have some serious issues - pushing for a reindex far, far too regularly (I have heard that it triggers off every new post!)
The plugin does trigger a reindex of the small delta index after every new post, to allow search results to be updated in real time. That is necessary since a lot of people navigate phpBB using only the "View your posts" link at the top which is in fact a search for posts by a certain author.
Arantor wrote: and stopping searchd repeatedly. Note that instances where searchd appears to be crashing are not - searchd is being forced to die with killall -9. See shutdown_searchd() in /sphinx_plugin/includes/search/fulltext_sphinx.php (called twice by create_index(), for example!)

This repeated process killing is the cause of 'no results' (since I've seen cases where it happens every 30 seconds!)
You should take a look under which conditions shutdown_searchd() is called at all. It's called when the pid file is missing to make sure, there is really no process running (this should never happen). It's also called after the configuration has been changed in the ACP which requires a searchd restart. It is called in create_index, however that function is only called if you manually initially create an index from the ACP, and never during regular use. It's also called when you manually delete an index in the ACP. So basically during regular use this function is _never_ called.

I have to admit however, that SIGKILL might be too much, most of the time SIGTERM should suffice. I will change that in a future version.

Arantor wrote:I will also add that we have discussed this plugin over there, and I was not entirely complimentary about it, and have actually said I'll build my own. I will be targetting phpBB3 and provide support for via the Sphinx forum.
I'd be happy to hear suggestions or apply patches, if you want to make changes to this plugin. There's no point in offering two different plugins. I could also add you as a regular developer to this project. I agree there's a lot this plugin doesn't do/does incorrectly I just don't have too much time to work on it.
I appreciate gifts from my Amazon wishlist.
naderman.de twitter: @naderman

latv
Registered User
Posts: 1
Joined: Mon Jun 01, 2009 7:54 pm

Re: [Beta] Sphinx search for phpBB 1.0.beta2

Post by latv » Mon Jun 01, 2009 8:09 pm

Thanx a lot, naderman, really great stuff, works perfectly despite some architectural issues discussed here.

Board searching, posting and reindexing became quick as a wink :mrgreen:

There was one problem at install time - I had to create phpbb_sphinx manually, guessing it's structure from SQL commands in sphinx' config.

Letus2001
Registered User
Posts: 22
Joined: Thu Nov 04, 2004 8:22 pm

Re: [Beta] Sphinx search for phpBB 1.0.beta2

Post by Letus2001 » Tue Jun 09, 2009 10:38 am

hi naderman, thanks for the reply. Do you have any hint how to solve my other problem with searching terms like

F-16
T-34

with the - in it ? I found in your code you do some kind of replacement, but im unable to disable it, so the search term stays unmodified, I need to be able to search for those terms without any modification, just as user typed them.

User avatar
naderman
Consultant
Consultant
Posts: 3735
Joined: Fri Aug 01, 2003 10:06 pm
Location: Berlin, Germany
Name: Nils Adermann
Contact:

Re: [Beta] Sphinx search for phpBB 1.0.beta2

Post by naderman » Tue Jun 09, 2009 3:44 pm

Do you know whether sphinx is capable of searching for such terms? I would think it splits them up as different words?
I appreciate gifts from my Amazon wishlist.
naderman.de twitter: @naderman

arod-1
Registered User
Posts: 1327
Joined: Mon Sep 20, 2004 1:33 pm

Re: [Beta] Sphinx search for phpBB 1.0.beta2

Post by arod-1 » Tue Jun 09, 2009 4:36 pm

from the manual:
sphinx-manual wrote: 6.7.4. EscapeString

Prototype: function EscapeString ( $string )

Escapes characters that are treated as special operators by the query language parser. Returns an escaped string.

$string is a string to escape.

This function might seem redundant because it's trivial to implement in any calling application. However, as the set of special characters might change over time, it makes sense to have an API call that is guaranteed to escape all such characters at all times.

Usage example:

$escaped = $cl->EscapeString ( "escaping-sample@query/string" );
btw: do you have plans to move the mod status from Beta?
thanks.
standard disclaimer:
backup your db and files before you do anything.
absolutely no guarantee.
if you do what i advise and it blows in your face, all you'll hear from me is: "ah... sorry, i guess"

User avatar
DoYouSpeakWak
Registered User
Posts: 2307
Joined: Fri Jul 25, 2008 1:32 pm
Location: Island of Wak-Wak
Name: Hans Lassen
Contact:

Re: [ALPHA] Sphinx for phpBB

Post by DoYouSpeakWak » Tue Jun 09, 2009 5:50 pm

naderman wrote:
idiotnesia wrote:Is this feature will be available in phpBB 3.2 ?
Right now it's a plugin you can easily add to your phpBB3.0 without any code changes. It will probably also be included in 3.2 by default, yes.
I just downloaded the sphinx from there website. It seems to require a dedicated server or at least ssh to type in the commands. Most people run phpbb on shared hosting without either. Making it defualt with getting around that problem seems like a strange idea. But am i missing something ?

Wak
Whatever you share comes back. Support the phpBB Communities
My Validated and Released Modifications
Offering paid phpBB help and System administrator services.

ToonArmy
Former Team Member
Posts: 4608
Joined: Sat Mar 06, 2004 5:29 pm
Location: Worcestershire, UK
Name: Chris Smith
Contact:

Re: [ALPHA] Sphinx for phpBB

Post by ToonArmy » Tue Jun 09, 2009 6:17 pm

DoYouSpeakWak wrote:
naderman wrote:
idiotnesia wrote:Is this feature will be available in phpBB 3.2 ?
Right now it's a plugin you can easily add to your phpBB3.0 without any code changes. It will probably also be included in 3.2 by default, yes.
I just downloaded the sphinx from there website. It seems to require a dedicated server or at least ssh to type in the commands. Most people run phpbb on shared hosting without either. Making it defualt with getting around that problem seems like a strange idea. But am i missing something ?

Wak
Keyword is included not enabled/used by default.
Chris SmithGitHub

User avatar
DoYouSpeakWak
Registered User
Posts: 2307
Joined: Fri Jul 25, 2008 1:32 pm
Location: Island of Wak-Wak
Name: Hans Lassen
Contact:

Re: [Beta] Sphinx search for phpBB 1.0.beta2

Post by DoYouSpeakWak » Tue Jun 09, 2009 6:23 pm

Im not into [?] but

included in 3.2 by default. Does that mean i can pick either one :)

why include something 95 %(or around that) of phpbb user cant use. The standad search engine in phpbb are slow on larger boards, i would love to see that updated instead of 3rd party seach engine that requires alot of extra stuff.
Last edited by camm15h on Tue Jun 09, 2009 7:38 pm, edited 1 time in total.
Reason: Removed expletive?
Whatever you share comes back. Support the phpBB Communities
My Validated and Released Modifications
Offering paid phpBB help and System administrator services.

arod-1
Registered User
Posts: 1327
Joined: Mon Sep 20, 2004 1:33 pm

Re: [Beta] Sphinx search for phpBB 1.0.beta2

Post by arod-1 » Tue Jun 09, 2009 6:39 pm

DoYouSpeakWak wrote:Does that mean i can pick either one :)
yes.
DoYouSpeakWak wrote:why include something 95 %(or around that) of phpbb user cant use..
for the 5% who *can* use it? (in reality it's probably much less than 5% of the boards, but those tend to be the larger ones, so if you think in terms of "end users" it may be more or less than 5% of board-users)
how about "included support for Oracle database"? surely much less than 5% of users need this.
i would estimate that the ratio of # of people who can use sphinx / # of people who can use Oracle is more than 100.
so should Oracle support be dropped? or FireBird? or... (you get the picture)
standard disclaimer:
backup your db and files before you do anything.
absolutely no guarantee.
if you do what i advise and it blows in your face, all you'll hear from me is: "ah... sorry, i guess"

ToonArmy
Former Team Member
Posts: 4608
Joined: Sat Mar 06, 2004 5:29 pm
Location: Worcestershire, UK
Name: Chris Smith
Contact:

Re: [Beta] Sphinx search for phpBB 1.0.beta2

Post by ToonArmy » Tue Jun 09, 2009 7:47 pm

DoYouSpeakWak wrote:Does that mean i can pick either one :)
Yes
DoYouSpeakWak wrote:why include something 95 %(or around that) of phpbb user cant use. The standad search engine in phpbb are slow on larger boards, i would love to see that updated instead of 3rd party seach engine that requires alot of extra stuff.
Remember this maybe 5% of our users but of our users users the figure is a much larger proportion. Whilst a search back end written in PHP with the scalability of sphinx would be nice, in PHP its a fairly difficult task (there are some, which we are looking at) and back ends written in C, C++, etc. will provide much more performance to those that need it. End of discussion, this is not the place. Feel free to bring it up in phpBB discussion if you wish.
Chris SmithGitHub

User avatar
naderman
Consultant
Consultant
Posts: 3735
Joined: Fri Aug 01, 2003 10:06 pm
Location: Berlin, Germany
Name: Nils Adermann
Contact:

Re: [Beta] Sphinx search for phpBB 1.0.beta2

Post by naderman » Tue Jun 09, 2009 8:04 pm

arod-1 wrote:from the manual:
sphinx-manual wrote: 6.7.4. EscapeString

Prototype: function EscapeString ( $string )

Escapes characters that are treated as special operators by the query language parser. Returns an escaped string.

$string is a string to escape.

This function might seem redundant because it's trivial to implement in any calling application. However, as the set of special characters might change over time, it makes sense to have an API call that is guaranteed to escape all such characters at all times.

Usage example:

$escaped = $cl->EscapeString ( "escaping-sample@query/string" );
btw: do you have plans to move the mod status from Beta?
thanks.
Maybe some day. Right now I'm working on some other search backend plugins since I'm not really happy with sphinx usability from an administration point of view.
I appreciate gifts from my Amazon wishlist.
naderman.de twitter: @naderman

arod-1
Registered User
Posts: 1327
Joined: Mon Sep 20, 2004 1:33 pm

Re: [Beta] Sphinx search for phpBB 1.0.beta2

Post by arod-1 » Tue Jun 09, 2009 8:56 pm

search plugin infrastructure is, imo, not correct.
what i mean is, the plugin has no effect on the UI, which makes it problematic: a plugin can not support some new feature that the "canonical" search does not support, and otoh, there is no way for the plugin to say: "hey, i do not support this feature".

so, for instance, mysql fulltext (and sphinx) can't advertise/utilize its support for phrase search (or proximity, or prioritization of hits etc.), and sphinx can't bail-off from "search in forum", so you end up with broken functionality on one hand, and brilliant functionality that can't be used on the other.

imo, the search plugin infrastructure should be reconsidered, so the plugin would be able to control some aspects of the UI, and some aspects of the language.

this, of course, is not trivial, but with the current implementation each plugin also requires a MOD to tweak the UI to fully support the backend engine capabilities and limitations.

this may be the wrong place to express those thoughts, but area51 is mostly dead, and at least this is relevant to sphinx.
standard disclaimer:
backup your db and files before you do anything.
absolutely no guarantee.
if you do what i advise and it blows in your face, all you'll hear from me is: "ah... sorry, i guess"

User avatar
naderman
Consultant
Consultant
Posts: 3735
Joined: Fri Aug 01, 2003 10:06 pm
Location: Berlin, Germany
Name: Nils Adermann
Contact:

Re: [Beta] Sphinx search for phpBB 1.0.beta2

Post by naderman » Tue Jun 09, 2009 9:13 pm

arod-1 wrote:search plugin infrastructure is, imo, not correct.
what i mean is, the plugin has no effect on the UI, which makes it problematic: a plugin can not support some new feature that the "canonical" search does not support, and otoh, there is no way for the plugin to say: "hey, i do not support this feature".
Yes, but that's kind of how it is right now. It's not like we're not aware of that.
arod-1 wrote:so, for instance, mysql fulltext (and sphinx) can't advertise/utilize its support for phrase search (or proximity, or prioritization of hits etc.), and sphinx can't bail-off from "search in forum", so you end up with broken functionality on one hand, and brilliant functionality that can't be used on the other.
Actually phrase search works just fine, the UI just doesn't explain it.
arod-1 wrote:this, of course, is not trivial, but with the current implementation each plugin also requires a MOD to tweak the UI to fully support the backend engine capabilities and limitations.
The only thing it requires a MOD for is modifying the UI which while it would certainly be nice is not strictly necessary to search.
arod-1 wrote:this may be the wrong place to express those thoughts, but area51 is mostly dead, and at least this is relevant to sphinx.
I don't see how this is relevant to Sphinx, this needs to be solved in phpBB so discuss it on area51, even though I don't see much point in discussing this. Please come up with a patch and submit it to the bugtracker.
I appreciate gifts from my Amazon wishlist.
naderman.de twitter: @naderman

arod-1
Registered User
Posts: 1327
Joined: Mon Sep 20, 2004 1:33 pm

Re: [Beta] Sphinx search for phpBB 1.0.beta2

Post by arod-1 » Tue Jun 09, 2009 9:39 pm

naderman wrote:The only thing it requires a MOD for is modifying the UI which while it would certainly be nice is not strictly necessary to search.
agree with the first part, but not with the second: if a feature is not represented in the UI, it can't be used by mortals, which in effect means it's not supported.
naderman wrote:I don't see how this is relevant to Sphinx
to some extent it is: sphinx supports some totally cool features that can't be used because of the plugin infrastructure, for instance its "relevance ranking": sphinx provides its hits in a descending order of relevance, but the "front-end" reorder them by date, completely losing this information. it would be very useful to let the user select "order by date/order by relevance" on the UI, but this is not possible.
naderman wrote:Please come up with a patch and submit it to the bugtracker.
i might be tempted, but i do not have good experience with phpbb team's acceptance of code from outsiders.

specifically, the mysql fulltext backend is seriously broken, in that it indexes twice as much as it needs to, which hampers performance (on post/edit. search performance is not affected) and disk space. i submitted in the past patches that fixed this which were completely ignored.

creating a patch like this is a lot of work, and if the perceived state of affairs is that it has small chances of being accepted, i do not wish to waste my time.

peace.
standard disclaimer:
backup your db and files before you do anything.
absolutely no guarantee.
if you do what i advise and it blows in your face, all you'll hear from me is: "ah... sorry, i guess"

Locked

Return to “[3.0.x] MODs in Development”