| home / programming / Lucene / 2 |   [next] |
|
|
|
Written by Otis Gospodnetic and Erik Hatcher and reproduced from "Lucene in Action" by permission of Manning Publications Co. ISBN 1932394281, copyright 2004. All rights reserved. See http://www.manning.com for more information.
As you saw in our Indexer class, you need the following classes to perform the simplest indexing procedure:
IndexWriterDirectoryAnalyzerDocumentFieldWhat follows is a brief overview of these classes, to give you a rough idea about their role in Lucene. We'll use these classes throughout this book.
IndexWriter is the central component of the indexing process. This class creates
a new index and adds documents to an existing index. You can think of IndexWriter as an object that gives you write access to the index but doesn’t let you read
or search it. Despite its name, IndexWriter isn’t the only class that’s used to modify
an index; section 2.2 describes how to use the Lucene API to modify an index.
The Directory class represents the location of a Lucene index. It’s an abstract
class that allows its subclasses (two of which are included in Lucene) to store the
index as they see fit. In our Indexer example, we used a path to an actual file system
directory to obtain an instance of Directory, which we passed to IndexWriter’s
constructor. IndexWriter then used one of the concrete Directory implementations,
FSDirectory, and created our index in a directory in the file system.
In your applications, you will most likely be storing a Lucene index on a disk.
To do so, use FSDirectory, a Directory subclass that maintains a list of real files
in the file system, as we did in Indexer.
The other implementation of Directory is a class called RAMDirectory.
Although it exposes an interface identical to that of FSDirectory, RAMDirectory
holds all its data in memory. This implementation is therefore useful for smaller
indices that can be fully loaded in memory and can be destroyed upon the termination
of an application. Because all data is held in the fast-access memory and
not on a slower hard disk, RAMDirectory is suitable for situations where you need
very quick access to the index, whether during indexing or searching. For
instance, Lucene’s developers make extensive use of RAMDirectory in all their
unit tests: When a test runs, a fast in-memory index is created or searched; and
when a test completes, the index is automatically destroyed, leaving no residuals
on the disk. Of course, the performance difference between RAMDirectory and
FSDirectory is less visible when Lucene is used on operating systems that cache
files in memory. You’ll see both Directory implementations used in code snippets
in this book.
Before text is indexed, it’s passed through an Analyzer. The Analyzer, specified
in the IndexWriter constructor, is in charge of extracting tokens out of text to be
indexed and eliminating the rest. If the content to be indexed isn’t plain text, it
should first be converted to it, as depicted in figure 2.1. Chapter 7 shows how to
extract text from the most common rich-media document formats. Analyzer is
an abstract class, but Lucene comes with several implementations of it. Some of
them deal with skipping stop words (frequently used words that don’t help distinguish
one document from the other, such as a, an, the, in, and on); some deal with
conversion of tokens to lowercase letters, so that searches aren’t case-sensitive;
and so on. Analyzers are an important part of Lucene and can be used for much
more than simple input filtering. For a developer integrating Lucene into an
application, the choice of analyzer(s) is a critical element of application design.
You’ll learn much more about them in chapter 4.
A Document represents a collection of fields. You can think of it as a virtual document—
a chunk of data, such as a web page, an email message, or a text file—
that you want to make retrievable at a later time. Fields of a document represent
the document or meta-data associated with that document. The original source
(such as a database record, a Word document, a chapter from a book, and so on)
of document data is irrelevant to Lucene. The meta-data such as author, title,
subject, date modified, and so on, are indexed and stored separately as fields of
a document.
NOTE When we refer to a document in this book, we mean a Microsoft Word,
RTF, PDF, or other type of a document; we aren’t talking about Lucene’s
Document class. Note the distinction in the case and font.
Lucene only deals with text. Lucene’s core does not itself handle anything but
java.lang.String and java.io.Reader. Although various types of documents can
be indexed and made searchable, processing them isn’t as straightforward as processing
purely textual content that can easily be converted to a String or Reader
Java type. You’ll learn more about handling nontext documents in chapter 7.
In our Indexer, we’re concerned with indexing text files. So, for each text file
we find, we create a new instance of the Document class, populate it with Fields
(described next), and add that Document to the index, effectively indexing the file.
Each Document in an index contains one or more named fields, embodied in a
class called Field. Each field corresponds to a piece of data that is either queried
against or retrieved from the index during search.
Lucene offers four different types of fields from which you can choose:
Keyword Isn't analyzed, but is indexed and stored in the index verbatim.
This type is suitable for fields whose original value should be preserved in
its entirety, such as URLs, file system paths, dates, personal names, Social
Security numbers, telephone numbers, and so on. For example, we used
the file system path in Indexer (listing 1.1) as a Keyword field.
UnIndexed Is neither analyzed nor indexed, but its value is stored in the
index as is. This type is suitable for fields that you need to display with
search results (such as a URL or database primary key), but whose values
you'll never search directly. Since the original value of a field of this type is
stored in the index, this type isn't suitable for storing fields with very large
values, if index size is an issue.
UnStored The opposite of UnIndexed. This field type is analyzed and
indexed but isn't stored in the index. It's suitable for indexing a large
amount of text that doesn't need to be retrieved in its original form, such
as bodies of web pages, or any other type of text document
Text Is analyzed, and is indexed. This implies that fields of this type can
be searched against, but be cautious about the field size. If the data
indexed is a String, it's also stored; but if the data (as in our Indexer example)
is from a Reader, it isn't stored. This is often a source of confusion, so
take note of this difference when using Field.Text.
All fields consist of a name and value pair. Which field type you should use
depends on how you want to use that field and its values. Strictly speaking,
Lucene has a single Field type: Fields are distinguished from each other based
on their characteristics. Some are analyzed, but others aren’t; some are indexed,
whereas others are stored verbatim; and so on.
Table 1.2 provides a summary of different field characteristics, showing you how fields are created, along with common usage examples.
Notice that all field types can be constructed with two Strings that represent the
field’s name and its value. In addition, a Keyword field can be passed both a
String and a Date object, and the Text field accepts a Reader object in addition to
the String. In all cases, the value is converted to a Reader before indexing; these
additional methods exist to provide a friendlier API.
Field.Text(String, String) and Field.
Text(String, Reader). The String variant stores the field data, whereas
the Reader variant does not. To index a String, but not store it, use
Field.UnStored(String, String).Finally, UnStored and Text fields can be used to create term vectors (an advanced
topic, covered in section 5.7). To instruct Lucene to create term vectors for a
given UnStored or Text field, you can use Field.UnStored(String, String, true),
Field.Text(String, String, true), or Field.Text(String, Reader, true).
You’ll apply this handful of classes most often when using Lucene for indexing. In order to implement basic search functionality, you need to be familiar with an equally small and simple set of Lucene search classes.
| home / programming / Lucene / 2 | [next] |
| ||||||||||||||||||||
Created: March 27, 2003
Revised: January 31, 2005
URL: http://webreference.com/programming/lucene/2
