#85: Colons and special characters in Ultrasphinx queries

Solved!

on February 19, 2010, 09:59 PM UTC — from 92049143cabb7ba896d7c06e19906303_small

yliu ( 292 views )

When I use UltraSphinx, the Ruby API to the Sphinx full-text search engine, queries with certain special characters such as colons (:), at signs (@), and other reserved characters throw an Ultrasphinx::UsageError exception in my query interface.

For example, a search for "http://foo.com" reports:
Ultrasphinx::UsageError: index delta,main: query error: no field 'http' found in schema
Different characters cause different variants of the "query error: no field" found in schema problem. Just silencing the error or replacing the colons is unsatisfying. I am interested in what is causing this problem, and how can I reliably escape these characters.

1 solution | 2 references

Add Solution Add Reference

Solutions

Solutions Feed

Two different problems causing Ultrasphinx query UsageError exception

- yliu on February 19, 2010, 10:17 PM UTC

In Sphinx extended query syntax, there are currently (as of this post) 9 special characters:
( ) | - ! @ ~ \" &
each of them can be escaped by prepending the classic escape character, backslash ('\'). There is an implementation in the PHP API for Sphinx called EscapeString(), which will do this escaping for query strings for you. I can't seem to find it in Ruby's Ultrasphinx, so I had to reimplement it. Trivial enough.

The one special case is the colon (:) character. It's not in the Sphinx query language, so looking for it there won't help. Ultrasphinx (as of the version current at time of this post) hard-codes this character as a delimiter of field operators. That is, given "foo:bar", it will always attempt to reparse it into "@foo bar", which in Sphinx query language means to search for the string "bar" in the indexed database field foo.

In ultrasphinx/lib/ultrasphinx/search/parser.rb:

   1  
   2        def token_stream_to_hash(token_stream)
   3          token_hash = Hash.new([])        
   4          token_stream.map do |operator, content|
   5            # Remove some spaces
   6            content.gsub!(/^"\s+|\s+"$/, '"')
   7            # Convert fields into sphinx style, reformat the stream object
   8            if content =~ /(^(http|https):\/\/[a-z0-9]+([-.]{1}[a-z0-9]*)+. [a-z]{2,5}(([0-9]{1,5})?\/.*)?$)/ix
   9              # XXX hack, its somewhat common to search for URLs.  be sure to add 
  10              # " @, /," in the charset_type of the US config to search on all 
  11              # URLs and email addresses, and add:
  12              # prefix_fields = url, domain
  13              # to your US config
  14              token_hash[nil] += [[operator, content]]
  15            elsif content =~ /(.*?):(.*)/
  16              token_hash[$1] += [[operator, $2]]
  17            else
  18              token_hash[nil] += [[operator, content]]
  19            end        
  20          end
  21          token_hash
  22        end

The line "elsif content =~ /(.*?):(.*)/" performs this hard-coded split. As you can see, no amount of escaping backslashes on your end is going to fix this. Since usually people don't index colons anyway, it doesn't really matter.

The naive approach to the problem is to take out the elsif clause and use standard Sphinx syntax ("@foo bar") for advanced field-based ("foo:bar") queries. However, there is no telling what other key functionality relies on this behavior, and you're basically lobotomizing the alternate advanced query syntax. The more intelligent way would be to implement escaping, probably via backslash. I leave that as an exercise to the reader (please post your solution if you have one).