Note: SWISH has changed considerably since version 1.0! The format is different and configuration variables have changed. Current SWISH users should reindex their files using version 1.1. Please read the list of changes since version 1.0.
For an example of swish can do, try searching for the words "office and map" at EIT. All of the search databases you see there (with the exception of the Directory of Servers) were indexed by swish. When you do a search, it's the swish program that's doing the actual searching.
SWISH was created to fill the need of the growing number of Web administrators on the Internet - many current indexing systems are not well documented, are hard to use and install, and are too complex for their own good. Here are some pros and cons regarding SWISH:
I've tried to make SWISH as simple as possible while keeping some of the things that people look for in an indexer. The drawback is that you can't do many things that full-featured indexers and searching programs can do, such as stemming (searching for different versions of a word) or the use of synonyms.
In indexing HTML files, SWISH can ignore data in tags and give higher relevance to information in header and title tags. Titles are extracted from HTML files and appear in the search results. SWISH can automatically search your whole Web site for you in one pass, if it's under one directory. You can also search for words that exist in HTML titles, comments, and emphasized tags, and 8-bit HTML characters can be indexed, converted, and searched.
Index files consist of only one file, so they can be transported around and easily maintained. The SWISH source is not large and generated indexes average out to around half the size of comparable WAIS indexes, or 1 to 5% of the size of your original HTML data. Searching is as fast as or better than using a non-commercial WAIS-based solution.
I encourage people to send in patches and suggestions on how to make SWISH better. Although it's not in the public domain, I am always more than happy to integrate contributed code into the distribution. Please note the license concerning its use. If you do have licensing questions, please contact Jay Weber at firstname.lastname@example.org.
ftp://ftp.eit.com/pub/web.software/swish/. You can also download the package from this page (swish.11.tar.Z, 145k). Here's what to do:
swish.11.tar.Z) on your computer
tar -xvf swish.11.tar)
READMEfile for compilation instructions
Everything was written in pretty vanilla C, so it should work just about anywhere. Initial testing was done on the following systems: SunOS 4.1.3, Solaris 2.4, BSDI 1.1, IRIX 5.3/4, and OSF/1 2.0.
The swish program can go under
/usr/local/bin - you may want to put other SWISH things somewhere such as
/usr/local/httpd/swish, if you're using NCSA's httpd. You'll also want to create a directory to hold SWISH databases, somewhere like
/usr/local/httpd/swish/sources. You can store the files anywhere you like, as long as you remember where they are!
After you've compiled (and installed) SWISH, make sure the swish program is somewhere in your executable path (somewhere such as
swish -f sample.swish -w internet and resources and archie
This will search the file sample.swish for files consisting of the words internet and resources and archie. You should get something back like this:
# SWISH format 1.1 search words: internet and resources and archie # Name: Index of EIT's Web # Saved as: sample.swish # Counts: 7316 words, 94 files # Indexed on: 12/03/95 17:50:43 PST # Description: This is a full index of EIT's web site. # Pointer: http://www.eit.com/cgi-bin/wwwwais/ # Maintained by: Kevin Hughes (email@example.com) 1000 http://www.eit.com/web/www.guide/guide.15.html "Guide to Cyberspace 6.1: Index/Glossary" 11566 360 http://www.eit.com/web/netservices.html "Internet Resources List" 48391 .
The results tell you:
If there are errors, instead of the results list, you may get one of the following error lines. These lines will always be prefixed with
err: no results
err: could not open index file
err: no search words specified
err: a word is too common
err: the index file is empty
err: the index file format is unknown
/usr/local/httpd/swish/. The configuration file below is an example of a typical SWISH configuration file:
# SWISH configuration file # Lines beginning with hash marks (#) and # blank lines are ignored. IndexDir /usr/local/www # This is the root directory of the Web tree you want to index. # It can be a space-separated list of files and directories # you want indexed. You can specify more than one of these directives. IndexFile /usr/local/httpd/swish/sources/index.swish # This is the name your SWISH index will be saved as. IndexOnly .html .txt .c .ps .gif .au .hqx .xbm .mpg .pict .tiff # Only files with these suffixes will be indexed. IndexReport 3 # This is how detailed you want reporting. You can specify numbers # 0 to 3 - 0 is totally silent, 3 is the most verbose. FollowSymLinks yes # Put "yes" to follow symbolic links in indexing, else "no". NoContents .ps .gif .au .hqx .xbm .mpg .pict .tiff # Files with these suffixes won't have their contents indexed, # only their file names. ReplaceRules replace "/usr/local/www" "http://www.eit.com" # ReplaceRules append "" # ReplaceRules prepend "" # ReplaceRules allow you to make changes to file pathnames # before they're indexed. FileRules pathname contains admin testing demo trash construction confidential FileRules filename is index.html FileRules filename contains # % ~ .bak .orig .old old. FileRules title contains construction example pointers FileRules directory contains .htaccess # Files matching the above criteria will *not* be indexed. IgnoreWords SwishDefault # The IgnoreWords option allows you to specify words to ignore (stopwords). # Comment out for no stopwords; the word "SwishDefault" will include # a list of default stopwords. Words should be separated by spaces # and may span multiple directives.
To index a site using the options in a configuration file, type:
swish -c /usr/local/httpd/swish/swish.conf
To run swish and index your site.
Taking as an example the above configuration in the script, you'd have the directory
/usr/local/httpd/swish/sources and one file called
index.swish in the directory. The name of the database you've just created is
Basic index variables
The IndexDir variable tells swish what directories and files to index. Each specified directory will be indexed recursively. You can use more than one of these directives - here are some examples:
IndexDir /usr/local/www /src/code.html IndexDir /users/tony/public_html/home.html /web
The IndexFile variable tell swish what to save the indexed results as. Indexes generated by swish should have a suffix of
.suffix1 .suffix2 .suffix3 ...
Only files with these suffixes will be indexed. If you omit this variable, swish will index every file it comes across. Suffix checking is not case sensitive.
This variable can have the values
3. If you specify
3, swish will tell you what's going on while it's indexing, printing out directory and file names, number of words indexed, and so on, as well as give information about other operations. The value
0 will make swish completely silent.
Normally swish ignores symbolic links to files whe indexing. If you want it to follow such links, define this value as
yes, else define it as
.suffix1 .suffix2 .suffix3 ...
This variable lets you control what files will have their contents indexed. If a file with a suffix in this list is indexed, only its file name (and not any words in the file) will be indexed. This is useful because normally SWISH will try to index the contents of every file, even files without words (such as images or movies). Suffix checking is case-insensitive.
word1 word2 ...
Here you can specify words to ignore when searching. Usually these words (called stopwords are words that occur too many times in your data to make indexing them worthwhile. If you specify a word as
SwishDefault, it will be replaced with swish's default list - a few hundred very common English words.
After indexing, swish can automatically tell which words are the most common and omit them from the index according to these parameters. Here are some examples:
1. IgnoreLimit 80 256 2. IgnoreLimit 50 50
Using IgnoreLimit and IgnoreWords can help trim the size of your index files considerably - experiment with parameters to see what works best at your site. You can also use IgnoreLimit to limit the CPU resources that searches take.
These variables specify information that goes into index files to help users and administrators. IndexName should be the name of your index, like a book title. IndexDescription is a short description of the index or a URL pointing to a more full description. IndexPointer should be a pointer to the original information, most likely a URL. IndexAdmin should be the name of the index maintainer and can include name and email information. These values should not be more than 70 or so characters and should be contained in quotes. Note that the automatically generated date in index files is in
D/M/Y and 24-hour format.
When results are returned from swish searches, you may get a bunch of funny pathnames to files that you can't access. Using ResultRules, you can specify a series of operations to perform on the pathname result to change it into a URL and other things if you desire.
There are three operations you can specify: replace, append, and prepend. They will parse the pathname in the order you've typed these commands. More than one command and its arguments can appear on the same line, but it's easier to read when commands are broken up over a few lines. You can't put a command and its argument(s) on different lines, however.
Here's the syntax:
replace "the string you want replaced" "what to change it to" This replaces all occurrences of the old string with the new one. prepend "a string to add before the result" append "a string to add after the result"
Study the above sample configuration file and try things out. You'll find that by having swish return URLs instead of pathnames, you can create interfaces to swish that can allow users to get to the search results over the World-Wide Web.
You can specify certain file directives in the configuration file - any files or directories matching these criteria will be ignored and will not be indexed. Append all of these operations with the FileRules directive:
string1 string2 string3 ...
Any path names containing exactly these strings, whether they be paths to directories or paths to files, will be ignored. Using this you can avoid indexing temporary directories or private material.
Any file name exactly matching the specified file name will be ignored (this is case-sensitive). This cannot be a path.
string1 string2 string3 ...
Any file name containing these strings will be ignored (this is not case-sensitive). This cannot be a path.
string1 string2 string3 ...
Any HTML file with a title that contains these strings will be ignored (this is case-insensitive).
string1 string2 string3 ...
Any directory that contains any of these specified file names will be ignored (this is case-insensitive).
usage: swish [-i dir file ... ] [-c file] [-f file] [-l] [-v (num)] swish -w word1 word2 ... [-f file1 file2 ...] [-m num] [-t str] swish -M index1 index2 ... outputfile swish -D file swish -V options: defaults are in brackets -i : create an index from the specified files -w : search for words "word1 word2 ..." -t : tags to search in - specify as a string "HBthec" - in head, body, title, header, emphasized, or comments -f : index file to create or search from [index.swish] -c : configuration file to use for indexing -v : verbosity level (0 to 3)  -l : follow symbolic links when indexing -m : the maximum number of results to return  -M : merges index files -D : decodes an index file -V : prints the current version version: 1.1 docs: http://www.eit.com/software/swish/
To see the usage, run swish with a -z or -? option.
index.swishin the current directory. You don't need to put quotes around search words.
You can use the booleans and, or, or not in searching. Without these booleans, swish will assume you're anding the words together. Evaluation takes place from left to right only, although you can use parentheses to force the order of evaluation.
You can also use wildcards (asterisks) to search for matches to the beginnings of words only - you can't put asterisks at the front or in the middle of words.
example 1: swish -w john and doe or jane example 2: swish -w john and (doe or not jane) example 3: swish -w not (john or jane) and doe example 4: swish -w j* and doe
john or janewill be evaluated first, a
notoperation will be performed on that, then everything will be
jand that also contain
<HEAD>tags, B stands for
<BODY>tags, t is all
<TITLE>tags, h is
<H6>(header) tags, e is emphasized tags (this may be
<STRONG>), and c is HTML comment tags (
<!-- ... -->).
example 1: swish -w apples oranges -t t example 2: swish -w keywords draft release -t c example 3: swish -w world wide web -t the
all, there will be no limit to the number of results. The configuration file value overrides this value.
If you specify a directory to index, an index file, or the verbose option on the command-line, these values will override any specified in the configuration file.
You can specify multiple configuration files in order to split up common preferences. For instance, you might store a file with the stopwords in it and have multiple other files that have different index file information.
example 1: swish -c swish.conf example 2: swish -i /usr/local/www -f index.swish -v -c swish.conf example 3: swish -c swish.conf stopwords.conf
swish.confwill be read, then the variable in
stopwords.confwill be read. Note that if the same variables occur in both files, older values may be written over.
index.swishin the current directory.
0for completely silent operation and
3for detailed reports. If no value is given then
The -V option makes swish spit out its version number.
You can use a FileRules operation to exclude the particular file name, or pathname, or its title. If there are serious problems in indexing certain types of files, they may not have valid text in them (they may be binary files, for instance). You can use NoContents to exclude that type of file.How do I allow users on the Web to search my indexes?
Good question. You will need a gateway CGI program that presents users with a search form and options, calls swish with these options, and returns the data to them in a nice HTML format. Swish is not meant to do this. One swish-compatible gateway you can currently use is WWWWAIS, available atI want to make my own gateway program.
Great! Good gateways can be made that take advantage of swish's features. If you do make one, even a simple one, please let me know and I can include it in the distribution.How can I index all my compressed files?
Swish doesn't currently have the capability to do on-the-fly filtering of files. In the meantime, first index the uncompressed data, compress it, and using a ReplaceRules operation, change the suffix of indexed files to .Z or whatever is appropriate. That way users can retrieve the compressed information.Can I index 8-bit text?
Yes, if the text uses the HTML equivalents for the ISO-Latin-1 (ISO8859-1) character set. Upon indexing swish will convert all numbered entities it finds (such asHow can I index phrases?
©) to named entities (such as
©). To search for words including these codes, type the named entity (if it exists) in place of the 8-bit character. Swish will also convert entities to ASCII equivalents, so words that might look like this in HTML:
resumécan be searched as this:
resume. Please read the README file included with the distribution for information on changing these options.
Currently the only way to do this is to use the HTML entityHow can I implement keywords in my documents?
(non-breaking space) to represent a space in your HTML. It will then be indexed with a space. To search for the phrase, you'd have to enter
to represent a space also.
In your HTML files you can put keywords in comments, such as:I want to generate a list of files to be indexed and pass it to swish.
<!-- keywords computer camera -->...then when you search, swish should be called with the -t c option, such as:
swish -t c -w keywords computerAll documents that contains the words keywords and computer in their comments will then be returned. Swish has an option in the source code that you can define to give more relevance to the words inside comments; if you're doing keywords in this fashion, you may want to use that option.
One thing you can do is make a simple script to generate a configuration file full of IndexDir directives. For instance, make a separate file calledI run out of memory trying to index my files.
files.confand put something like this in it:
IndexDir /this_is_file_1/file.html IndexDir /usr/local/www IndexDir file2.html /some/directory/ ...Then call swish like this (assuming you're using a main
swish -c swish.conf files.conf
It's true that indexing can take up a lot of memory! One thing you can do is make many indices of smaller content instead of trying to do everything at once. You can then merge all the smaller pieces together.What other features are planned?
These are things that are highly dependent on how busy I get. For one, the parser could stand improvements. I have also been thinking about incorporating proximity, so that you can search for words that are close together or far apart. I would like to possibly incorporate stemming and soundex matches, but I need to do some research in this area. The ability to filter files is also on the list, as is the ability to run as a distributed server.
I know how to implement custom fields, using comments - this would require some changes to the parser and this feature hasn't been implemented in this version due to time constraints.
I would very much like to make swish aware of the other meta-indexing programs out there (such as Harvest), so it can be used as a drop-in search engine. If you have information about doing this, please let me know.
As always, patches, improvements, suggestions, and corrections are gratefully accepted. Send 'em all to Kevin Hughes at firstname.lastname@example.org.