Class URLClassifyProcessor

  • All Implemented Interfaces:
    Closeable, AutoCloseable

    public class URLClassifyProcessor
    extends UpdateRequestProcessor
    Update processor which examines a URL and outputs to various other fields characteristics of that URL, including length, number of path levels, whether it is a top level URL (levels==0), whether it looks like a landing/index page, a canonical representation of the URL (e.g. stripping index.html), the domain and path parts of the URL etc.

    This processor is intended used in connection with processing web resources, and helping to produce values which may be used for boosting or filtering later.

    In the example configuration below, we construct a custom updateRequestProcessorChain and then instruct the /update requesthandler to use it for every incoming document.

     <updateRequestProcessorChain name="urlProcessor">
       <processor class="org.apache.solr.update.processor.URLClassifyProcessorFactory">
         <bool name="enabled">true</bool>
         <str name="inputField">id</str>
         <str name="domainOutputField">hostname</str>
       <processor class="solr.RunUpdateProcessorFactory" />
     <requestHandler name="/update" class="solr.UpdateRequestHandler">
     <lst name="defaults">
     <str name="update.chain">urlProcessor</str>

    Then, at index time, Solr will look at the id field value and extract it's domain portion into a new hostname field. By default, the following fields will also be added:

    • url_length
    • url_levels
    • url_toplevel
    • url_landingpage

    For example, adding the following document

     { "id":"" }

    will result in this document in Solr:

    • Method Detail

      • getCanonicalUrl

        public URL getCanonicalUrl​(URL url)
                            throws MalformedURLException
        Gets a canonical form of the URL for use as main URL
        url - The input url
        The URL object representing the canonical URL
      • length

        public int length​(URL url)
        Calculates the length of the URL in characters
        url - The input URL
        the length of the URL
      • levels

        public int levels​(URL url)
        Calculates the number of path levels in the given URL
        url - The input URL
        the number of levels, where a top-level URL is 0
      • isTopLevelPage

        public boolean isTopLevelPage​(URL url)
        Calculates whether a URL is a top level page
        url - The input URL
        true if page is a top level page
      • isLandingPage

        public boolean isLandingPage​(URL url)
        Calculates whether the URL is a landing page or not
        url - The input URL
        true if URL represents a landing page (index page)
      • isEnabled

        public boolean isEnabled()
      • setEnabled

        public void setEnabled​(boolean enabled)