Package org.apache.solr.cli

Class PostTool

  • All Implemented Interfaces:
    Tool
    public class PostTool
extends ToolBase

    • Constructor Detail

      • PostTool

        public PostTool()

    • Method Detail

      • getName

        public String getName()
        Description copied from interface: Tool
        Defines the interface to a Solr tool that can be run from this command-line app.

      • getOptions

        public List<org.apache.commons.cli.Option> getOptions()

      • execute

        public void execute()
             throws org.apache.solr.client.solrj.SolrServerException,
                    IOException
        After initialization, call execute to start the post job. This method delegates to the correct mode method.
        Throws:
        org.apache.solr.client.solrj.SolrServerException
        IOException

      • postFiles

        public int postFiles​(String[] args,
                     int startIndexInArgs,
                     OutputStream out,
                     String type)
        Post all filenames provided in args
        Parameters:
        args - array of file names
        startIndexInArgs - offset to start
        out - output stream to post data to
        type - default content-type to use when posting (may be overridden in auto mode)
        Returns:
        number of files posted

      • postWebPages

        public int postWebPages​(String[] args,
                        int startIndexInArgs,
                        OutputStream out)
        This method takes as input a list of start URL strings for crawling, converts the URL strings to URI strings and adds each one to a backlog and then starts crawling
        Parameters:
        args - the raw input args from main()
        startIndexInArgs - offset for where to start
        out - outputStream to write results to
        Returns:
        the number of web pages posted

      • normalizeUrlEnding

        protected static String normalizeUrlEnding​(String link)
        Normalizes a URL string by removing anchor part and trailing slash
        Returns:
        the normalized URL string

      • webCrawl

        protected int webCrawl​(int level,
                       OutputStream out)
        A very simple crawler, pulling URLs to fetch from a backlog and then recurses N levels deep if recursive>0. Links are parsed from HTML through first getting an XHTML version using SolrCell with extractOnly, and followed if they are local. The crawler pauses for a default delay of 10 seconds between each fetch, this can be configured in the delay variable. This is only meant for test purposes, as it does not respect robots or anything else fancy :)
        Parameters:
        level - which level to crawl
        out - output stream to write to
        Returns:
        number of pages crawled on this level and below

      • computeFullUrl

        protected String computeFullUrl​(URL baseUrl,
                                String link)
        Computes the full URL based on a base url and a possibly relative link found in the href param of an HTML anchor.
        Parameters:
        baseUrl - the base url from where the link was found
        link - the absolute or relative link
        Returns:
        the string version of the full URL

      • typeSupported

        protected boolean typeSupported​(String type)
        Uses the mime-type map to reverse lookup whether the file ending for our type is supported by the fileTypes option
        Parameters:
        type - what content-type to lookup
        Returns:
        true if this is a supported content type

      • commit

        public void commit()
            throws IOException,
                   org.apache.solr.client.solrj.SolrServerException
        Does a simple commit operation
        Throws:
        IOException
        org.apache.solr.client.solrj.SolrServerException

      • optimize

        public void optimize()
              throws IOException,
                     org.apache.solr.client.solrj.SolrServerException
        Does a simple optimize operation
        Throws:
        IOException
        org.apache.solr.client.solrj.SolrServerException

      • appendParam

        public static String appendParam​(String url,
                                 String param)
        Appends a URL query parameter to a URL
        Parameters:
        url - the original URL
        param - the parameter(s) to append, separated by "&"
        Returns:
        the string version of the resulting URL

      • guessType

        protected static String guessType​(File file)
        Guesses the type of file, based on file name suffix Returns "application/octet-stream" if no corresponding mimeMap type.
        Parameters:
        file - the file
        Returns:
        the content-type guessed

      • postData

        public boolean postData​(InputStream data,
                        Long length,
                        OutputStream output,
                        String type,
                        URL url)
        Reads data from the data stream and posts it to solr, writes to the response to output
        Returns:
        true if success

      • stringToStream

        public static InputStream stringToStream​(String s)
        Converts a string to an input stream
        Parameters:
        s - the string
        Returns:
        the input stream

      • getFileFilterFromFileTypes

        public FileFilter getFileFilterFromFileTypes​(String fileTypes)

      • getXP

        public static String getXP​(Node n,
                           String xpath,
                           boolean concatAll)
                    throws XPathExpressionException
        Gets the string content of the matching an XPath
        Parameters:
        n - the node (or doc)
        xpath - the xpath string
        concatAll - if true, text from all matching nodes will be concatenated, else only the first returned
        Throws:
        XPathExpressionException