Converts the query string to a Collection of Lucene tokens using a regular expression.
 Boolean operators AND, OR, NOT are skipped. 
 
 Each term is checked to determine if it is optional, required or prohibited.  Required
 terms output a 
Token with the 
QueryConverter.REQUIRED_TERM_FLAG set.
 Prohibited terms output a 
Token with the 
QueryConverter.PROHIBITED_TERM_FLAG 
 set. If the query uses the plus (+) and minus (-) to denote required and prohibited, this
 determination will be accurate.  In the case boolean AND/OR/NOTs are used, this
 converter makes an uninformed guess as to whether the term would likely behave as if it
 is Required or Prohibited and sets the flags accordingly.  These flags are used downstream
 to generate collations for 
WordBreakSolrSpellChecker, in cases where an original 
 term is split up into multiple Tokens.