From the point of view of stackoveflow.com (a team of 2-3 engineers creating a website project designed to scale), it makes sense to spend effort at an early stage of the development process to build a search based on Lucene / Autonomy ... as opposed to full-text search based on the database.
Pros / Cons:
With a mature Lucene implementation such as nutch or autonomy, the cost of switching to Lucene (which is inevitable) at a later stage is negligible.
In large volumes, adding additional index servers (say, with nutch) to maintain a growing search index is relatively easy.
When implementing Lucene IL, you will most likely need an additional server for the main memory index (at the very early stage of the scaling process).
source
share