After considering Lucene, I built my in house search engine. I wanted it to work a lot more like google than a "Full-text" library like search engine.
Very few users will go beyond the basics. How many users will actually used Google advanced search? Even programmers? Very few. Users don't use advanced search features.
Why? It's not their fault. They tried. They tried at the library - didn't work. They tried on the early search engines - didn't work. They tried on your Intranet app powered by database full text search - it doesn't work.
We trained them. We showed them that (most) advanced searching isn't worth their time.
Why is? Revisioning + speed. Make it easy to try different combos of search terms really fast. Correct spelling, suggest searches, add the ability to filter information.
See: Information Architecture for the World Wide Web p. 185.
If you're using Lucene, it's not too difficult to write your own query parser that does exactly what you want, including fuzzy searches, spelling correction, term weighting, etc. My current employer has a rather involved one that can take advantage of lots of application specific data.
Writing a custom query parser allows you to do as much hidden magic on the query as you want, and expose your own special operators if you really want to. Check out some of our user accessible operators at http://help.greplin.com/customer/portal/articles/15527-how-c...
Incidentally, I'd love to exchange notes and hear about your your search engine sometime. If you're interested, shoot me an email (address in profile).
After considering Lucene, I built my in house search engine.
Yeah, totally makes sense to write your own search engine instead.
Or you could use Lucene and have a tiny bit of code in front of it that tacks on "~" to query terms first and save yourself some time. Or spend a few minutes reading the Lucene docs and find out how to do what you'd like to do. I've had the misfortune of dealing with homegrown search engines...it was laughable to see all that work (requiring a bunch of heavy-duty servers, lots of development time) replaced with a Lucene index that took half a day to get up and running with much better performance+features on a tiny VPS.
A general idea is to 'compile' user queries to a lucene query with the various default options that give the best search, and not to use raw lucene queries from the user. The knobs need to be there for turning, whether you expose those knobs to the end user is a design decision.
We recently chose a Solr/Lucene solution for our game search at Big Fish Games (launches tomorrow!). I cannot imagine writing from scratch many features that come with Solr/Lucene: spelling correction, word stemming (walk == walking), stop words (don't return every hit for "the"), sane defaults for tokenizing (splitting up sentences into indexable and searchable chunks, like words), and uhh soon Fuzzy Query matching.
It's certainly easy to use a very limited subset of Lucene's capabilities to come up with a very intuitive user-searchable index of data.
If you wrote something from scratch that truly works better than an out of the box Solr server, let's just say I'd be surprised.
If you can't imagine writing a spelling corrector from scratch you might want to take a minute and broaden your horizons(I know it was magic pixie dust for me before I read it). http://norvig.com/spell-correct.html
Stemming could be accomplished just as easily, put in a word get back a list of stems. Just a dictionary look up that could be precomputed.
I thought his comment was more along the lines of "I cannot imagine writing industrial-strength versions of all of these things Lucene gives me."
Yes, you can write a spellchecker in 21 lines of code, that doesn't necessarily mean it will be fast enough to be a component in website search or that it will be any kind of a pleasure to query or maintain the word corpus.
I can put together toy versions of many things Lucene provides pretty easily in my own time. Building useful, dependable versions of most anything takes a nontrivial amount of time and effort, so it's smart to restrict my usage of toys to understanding the concepts.
I don't want to sound like a troll, but Lucene's built-in spellchecker is a toy. If you're doing much more than trying to just plonk in a search engine on your website, you'll probably find it insufficient.
As far as stemming goes, AFAIK, Lucene didn't invent anything. It's a lot like crypto, in that nearly everyone uses public implementations of of stemmers for different languages: http://snowball.tartarus.org/
The previous two points both apply to stop words (you'll almost certainly want to customize your own list, and there are good, curated lists out there for the most common languages), and Lucene's tokenizer options aren't really that smart -- if you want to get into chunking, phrase identification, etc., you're going to have to write it yourself. But then you're going to be working within the Lucene world, tied to their index format, mucking around in the depths of unfamiliar code, etc. That's a big trade-off.
Basically, Lucene is great if you're working on a website and you need a drop-in solution for search, but if your core user experience is search, it has a lot of deficiencies.
Are you saying you wrote an alternative to Lucene because users don't like to type in search queries that Lucene understands? Lucene is a low-level library for implementing a search experience. If you don't understand that then I don't think you understand how a sophisticated search engine like Google works.
Say you type in:
"Lucene equivalent in C++ or C#"
This query is not passed as-is to some magic implementation of a Search Engine by Google.
First they need to parse this query. This involves a tokenizer step which will have many rules depending on the context and the language requirements, etc. An industrial strength tokenizer is a significant undertaking in itself.
The next step is turn this user query into something that a search engine understands. Unless your default mode of operation is "match this user query EXACTLY" then that means having some kind of query language. For any search engine, this will end up looking like the syntax you have a problem with in your post.
Next, to have any kind of useful and competitive search experience you will need to take the user query and generates tens, maybe even hundreds or thousands, of different sub-queries that try to explore different estimates of user intent and fuzzy-matching. For example, you might do some analysis and realise that equivalent in this context is not very important and decide to generate another twenty queries with all-known synonyms for the word equivalent. You might decide to try the query with certain words dropped altogether.
And of course, you also need to be able to rank results. Not just within sub-queries but when merging all the results from sub-queries together. This takes sophisticated and highly customised scoring algorithms that will be dependent on context and a whole lot of other things. Essentially you need to guess what is the best match based on what you think the user wants.
Finally, to be an industry leader like Google you will use a data-driven approach to scoring that makes use of machine-learning techniques to keep your advantage more than a purely technology one.
For companies that just want a basic search that will behave intuitively for people used to the Google model, Lucene helps with A LOT of this. Their default query parser, scorer and other vital components are good enough that most websites can just use the default configurations out of the box and get a good search experience.
All that you're left with in the end is to create the basic user interface abstraction - the "one box" experience. Solr helps enormously with this, but in the end all the functionality it exposes is Lucene.
For more complicated search experiences, there isn't much that I described in the above approach that you can't do in Lucene.
If you are a PHP dev you don't need to bother wasting your time with a wrapper on top of Lucene. There's a very nice implementation called ElasticSearch which already has most of the functionality you'd be seeking.
Can you tell us more about your search engine? I'm building an online store and search is going to be crucial for us. Currently, I'm evaluating Lucene (solr).
The QueryParser syntax is term~ or term~N?
After considering Lucene, I built my in house search engine. I wanted it to work a lot more like google than a "Full-text" library like search engine.
Very few users will go beyond the basics. How many users will actually used Google advanced search? Even programmers? Very few. Users don't use advanced search features.
Why? It's not their fault. They tried. They tried at the library - didn't work. They tried on the early search engines - didn't work. They tried on your Intranet app powered by database full text search - it doesn't work.
We trained them. We showed them that (most) advanced searching isn't worth their time.
Why is? Revisioning + speed. Make it easy to try different combos of search terms really fast. Correct spelling, suggest searches, add the ability to filter information.
See: Information Architecture for the World Wide Web p. 185.
Also available on Safari Books.
See Also: Google