Package org.apache.solr.cli

Class PostTool

java.lang.Object

org.apache.solr.cli.ToolBase

org.apache.solr.cli.PostTool

All Implemented Interfaces:

Tool

public class PostTool
extends ToolBase

Supports post command in the bin/solr script.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	PostTool.PageFetcherResult	Utility class to hold the result form a page fetch

Field Summary

Fields

Modifier and Type	Field	Description
static final String	DEFAULT_CONTENT_TYPE
static final String	DEFAULT_FILE_TYPES

Fields inherited from class org.apache.solr.cli.ToolBase

runtime

Constructor Summary

Constructors

Constructor	Description
PostTool(ToolRuntime runtime)

Method Summary

Modifier and Type	Method	Description
static String	appendParam(String url, String param)	Appends a URL query parameter to a URL
protected static URI	appendUrlPath(URI uri, String append)	Appends to the path of the URL
void	commit()	Does a simple commit operation
protected static String	computeFullUrl(URL baseUrl, String link)	Computes the full URL based on a base url and a possibly relative link found in the href param of an HTML anchor.
void	execute(String mode)	After initialization, call execute to start the post job.
Predicate<Path>	getFileFilterFromFileTypes(String fileTypes)
String	getName()	Defines the interface to a Solr tool that can be run from this command-line app.
static NodeList	getNodesFromXP(Node n, String xpath)	Gets all nodes matching an XPath
org.apache.commons.cli.Options	getOptions()	Retrieve the Options supported by this tool.
static String	getXP(Node n, String xpath, boolean concatAll)	Gets the string content of the matching an XPath
protected static String	guessType(Path path)	Guesses the type of file, based on file name suffix Returns "application/octet-stream" if no corresponding mimeMap type.
static Document	makeDom(byte[] in)	Takes a string as input and returns a DOM
protected static String	normalizeUrlEnding(String link)	Normalizes a URL string by removing anchor part and trailing slash
void	optimize()	Does a simple optimize operation
boolean	postData(InputStream data, Long length, OutputStream output, String type, URI uri)	Reads data from the data stream and posts it to solr, writes to the response to output
void	postFile(Path file, OutputStream output, String type)	Opens the file and posts its contents to the solrUrl, writes to response to output.
int	postFiles(String[] args, int startIndexInArgs, OutputStream out, String type)	Post all filenames provided in args
int	postWebPages(String[] args, int startIndexInArgs, OutputStream out)	This method takes as input a list of start URL strings for crawling, converts the URL strings to URI strings and adds each one to a backlog and then starts crawling
void	runImpl(org.apache.commons.cli.CommandLine cli)
static InputStream	stringToStream(String s)	Converts a string to an input stream
protected boolean	typeSupported(String type)	Uses the mime-type map to reverse lookup whether the file ending for our type is supported by the fileTypes option
protected int	webCrawl(int level, OutputStream out)	A very simple crawler, pulling URLs to fetch from a backlog and then recurses N levels deep if recursive>0.

Methods inherited from class org.apache.solr.cli.ToolBase

echo, echoIfVerbose, getConnectionOptions, getRuntime, isVerbose, runTool

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.solr.cli.Tool

getFooter, getHeader, getUsage

Field Details

DEFAULT_FILE_TYPES

public static final String DEFAULT_FILE_TYPES

See Also:

• Constant Field Values

DEFAULT_CONTENT_TYPE

public static final String DEFAULT_CONTENT_TYPE

See Also:

• Constant Field Values

Constructor Details

PostTool

public PostTool(ToolRuntime runtime)

Method Details

getName

public String getName()

Description copied from interface: Tool

Defines the interface to a Solr tool that can be run from this command-line app.

getOptions

public org.apache.commons.cli.Options getOptions()

Description copied from interface: Tool

Retrieve the Options supported by this tool.

Specified by:

getOptions in interface Tool

Overrides:

getOptions in class ToolBase

Returns:

The Options this tool supports.

runImpl

public void runImpl(org.apache.commons.cli.CommandLine cli)
             throws Exception

Specified by:

runImpl in class ToolBase

Throws:

Exception

execute

public void execute(String mode)
             throws org.apache.solr.client.solrj.SolrServerException,
IOException

After initialization, call execute to start the post job. This method delegates to the correct mode method.

Throws:

org.apache.solr.client.solrj.SolrServerException

IOException

postFiles

public int postFiles(String[] args,
 int startIndexInArgs,
 OutputStream out,
 String type)
              throws IOException

Post all filenames provided in args

Parameters:

args - array of file names

startIndexInArgs - offset to start

out - output stream to post data to

type - default content-type to use when posting (this may be overridden in auto mode)

Returns:

number of files posted

Throws:

IOException - if an I/O error occurs

postWebPages

public int postWebPages(String[] args,
 int startIndexInArgs,
 OutputStream out)

This method takes as input a list of start URL strings for crawling, converts the URL strings to URI strings and adds each one to a backlog and then starts crawling

Parameters:

args - the raw input args from main()

startIndexInArgs - offset for where to start

out - outputStream to write results to

Returns:

the number of web pages posted

normalizeUrlEnding

protected static String normalizeUrlEnding(String link)

Normalizes a URL string by removing anchor part and trailing slash

Returns:

the normalized URL string

webCrawl

protected int webCrawl(int level,
 OutputStream out)

A very simple crawler, pulling URLs to fetch from a backlog and then recurses N levels deep if recursive>0. Links are parsed from HTML through first getting an XHTML version using SolrCell with extractOnly, and followed if they are local. The crawler pauses for a default delay of 10 seconds between each fetch, this can be configured in the delay variable. This is only meant for test purposes, as it does not respect robots or anything else fancy :)

Parameters:

level - which level to crawl

out - output stream to write to

Returns:

number of pages crawled on this level and below

computeFullUrl

protected static String computeFullUrl(URL baseUrl,
 String link)
                                throws MalformedURLException,
URISyntaxException

Computes the full URL based on a base url and a possibly relative link found in the href param of an HTML anchor.

Parameters:

baseUrl - the base url from where the link was found

link - the absolute or relative link

Returns:

the string version of the full URL

Throws:

MalformedURLException

URISyntaxException

typeSupported

protected boolean typeSupported(String type)

Uses the mime-type map to reverse lookup whether the file ending for our type is supported by the fileTypes option

Parameters:

type - what content-type to lookup

Returns:

true if this is a supported content type

commit

public void commit()
            throws IOException,
org.apache.solr.client.solrj.SolrServerException

Does a simple commit operation

Throws:

IOException

org.apache.solr.client.solrj.SolrServerException

optimize

public void optimize()
              throws IOException,
org.apache.solr.client.solrj.SolrServerException

Does a simple optimize operation

Throws:

IOException

org.apache.solr.client.solrj.SolrServerException

appendParam

public static String appendParam(String url,
 String param)

Appends a URL query parameter to a URL

Parameters:

url - the original URL

param - the parameter(s) to append, separated by "&"

Returns:

the string version of the resulting URL

postFile

public void postFile(Path file,
 OutputStream output,
 String type)
              throws MalformedURLException,
URISyntaxException

Opens the file and posts its contents to the solrUrl, writes to response to output.

Throws:

MalformedURLException

URISyntaxException

appendUrlPath

protected static URI appendUrlPath(URI uri,
 String append)

Appends to the path of the URL

Parameters:

uri - the URI

append - the path to append

Returns:

the final URL version

guessType

protected static String guessType(Path path)

Guesses the type of file, based on file name suffix Returns "application/octet-stream" if no corresponding mimeMap type.

Parameters:

path - path to the file

Returns:

the content-type guessed

postData

public boolean postData(InputStream data,
 Long length,
 OutputStream output,
 String type,
 URI uri)

Reads data from the data stream and posts it to solr, writes to the response to output

Returns:

true if success

stringToStream

public static InputStream stringToStream(String s)

Converts a string to an input stream

Parameters:

s - the string

Returns:

the input stream

getFileFilterFromFileTypes

public Predicate<Path> getFileFilterFromFileTypes(String fileTypes)

getNodesFromXP

public static NodeList getNodesFromXP(Node n,
 String xpath)
                               throws XPathExpressionException

Gets all nodes matching an XPath

Throws:

XPathExpressionException

getXP

public static String getXP(Node n,
 String xpath,
 boolean concatAll)
                    throws XPathExpressionException

Gets the string content of the matching an XPath

Parameters:

n - the node (or doc)

xpath - the xpath string

concatAll - if true, text from all matching nodes will be concatenated, else only the first returned

Throws:

XPathExpressionException

makeDom

public static Document makeDom(byte[] in)
                        throws SAXException,
IOException,
ParserConfigurationException

Takes a string as input and returns a DOM

Throws:

SAXException

IOException

ParserConfigurationException