Statistics
This section of the user guide covers the core statistical functions available in math expressions.
Descriptive Statistics
The describe
function returns descriptive statistics for a
numeric array. The describe
function returns a single tuple with name/value
pairs containing the descriptive statistics.
Below is a simple example that selects a random sample of documents from the logs collection,
vectorizes the response_d
field in the result set and uses the describe
function to
return descriptive statistics about the vector.
let(a=random(logs, q="*:*", fl="response_d", rows="50000"),
b=col(a, response_d),
c=describe(b))
When this expression is sent to the /stream
handler it responds with:
{
"resultset": {
"docs": [
{
"sumsq": 36674200601.78738,
"max": 1068.854686837548,
"var": 1957.9752647562789,
"geometricMean": 854.1445499569674,
"sum": 42764648.83319176,
"kurtosis": 0.013189848821424377,
"N": 50000,
"min": 656.023249311864,
"mean": 855.2929766638425,
"popVar": 1957.936105250984,
"skewness": 0.0014560741802307174,
"stdev": 44.24901428005237
},
{
"EOF": true,
"RESPONSE_TIME": 430
}
]
}
}
Notice that the random sample contains 50,000 records and the response time is only 430 milliseconds. Samples of this size can be used to reliably estimate the statistics for very large underlying data sets with subsecond performance.
The describe
function can also be visualized in a table with ZeppelinSolr:
Histograms and Frequency Tables
Histograms and frequency tables are tools for visualizing the distribution of a random variable.
The hist
function creates a histogram designed for usage with continuous data. The
freqTable
function creates a frequency table for use with discrete data.
histograms
In the example below a histogram is used to visualize a random sample of
response times from the logs collection. The example retrieves the
random sample with the random
function and creates a vector from the response_d
field
in the result set. Then the hist
function is applied to the vector
to return a histogram with 22 bins. The hist
function returns a
list of tuples with summary statistics for each bin.
let(a=random(logs, q="*:*", fl="response_d", rows="50000"),
b=col(a, response_d),
c=hist(b, 22))
When this expression is sent to the /stream
handler it responds with:
{
"resultset": {
"docs": [
{
"prob": 0.00004896007228311655,
"min": 675.573084576817,
"max": 688.3309631697003,
"mean": 683.805542728906,
"var": 50.9974629924082,
"cumProb": 0.000030022417162809913,
"sum": 2051.416628186718,
"stdev": 7.141250800273591,
"N": 3
},
{
"prob": 0.00029607514624062624,
"min": 696.2875238591652,
"max": 707.9706315779541,
"mean": 702.1110569558929,
"var": 14.136444379466969,
"cumProb": 0.00022705264963879807,
"sum": 11233.776911294284,
"stdev": 3.759846323916307,
"N": 16
},
{
"prob": 0.0011491235433157194,
"min": 709.1574910598678,
"max": 724.9027194369135,
"mean": 717.8554290699951,
"var": 20.6935845290122,
"cumProb": 0.0009858515418689757,
"sum": 41635.61488605971,
"stdev": 4.549020172412098,
"N": 58
},
...
]}}
With ZeppelinSolr the histogram can be first visualized as a table:
Then the histogram can be visualized with an area chart by plotting the mean of the bins on the xaxis and the prob (probability) on the yaxis:
The cumulative probability can be plotted by switching the yaxis to the cumProb column:
Custom Histograms
Custom histograms can be defined and visualized by combining the output from multiple
stats
functions into a single histogram. Instead of automatically binning a numeric
field the custom histogram allows for comparison of bins based on queries.
The stats function is first discussed in the Searching, Sampling and Aggregation section of the
user guide.

A simple example will illustrate how to define and visualize a custom histogram.
In this example, three stats
functions are wrapped in a plist
function. The
plist
(parallel list) function executes each of its internal functions in parallel
and concatenates the results into a single stream. plist
also maintains the order
of the outputs from each of the subfunctions. In this example each stats
function
computes the count of documents that match a specific query. In this case they count the
number of documents that contain the terms copper, gold and silver. The list of tuples
with the counts is then stored in variable a.
Then an array
of labels is created and set to variable l.
Finally the zplot
function is used to plot the labels vector and the count(*)
column.
Notice the col
function is used inside of the zplot
function to extract the
counts from the stats
results.
Frequency Tables
The freqTable
function returns a frequency distribution for a discrete data set.
The freqTable
function doesn’t create bins like the histogram. Instead it counts
the occurrence of each discrete data value and returns a list of tuples with the
frequency statistics for each value.
Below is an example of a frequency table built from a result set of rounded differences in daily opening stock prices for the stock ticker amzn.
This example is interesting because it shows a multistep process to arrive
at the result. The first step is to search for records in the the stocks
collection with a ticker of amzn. Notice that the result set is sorted by
date ascending and it returns the open_d
field which is the opening price for
the day.
The open_d
field is then vectorized and set to variable b, which now contains
a vector of opening prices ordered by date ascending.
The diff
function is then used to calculate the first difference for the
vector of opening prices. The first difference simply subtracts the previous value
from each value in the array. This will provide an array of price differences
for each day which will show daily change in opening price.
Then the round
function is used to round the price differences to the nearest
integer to create a vector of discrete values. The round
function in this
example is effectively binning continuous data at integer boundaries.
Finally the freqTable
function is run on the discrete values to calculate
the frequency table.
let(a=search(stocks,
q="ticker_s:amzn",
fl="open_d, date_dt",
sort="date_dt asc",
rows=25000),
b=col(a, open_d),
c=diff(b),
d=round(c),
e=freqTable(d))
When this expression is sent to the /stream
handler it responds with:
{
"resultset": {
"docs": [
{
"pct": 0.00019409937888198756,
"count": 1,
"cumFreq": 1,
"cumPct": 0.00019409937888198756,
"value": 57
},
{
"pct": 0.00019409937888198756,
"count": 1,
"cumFreq": 2,
"cumPct": 0.00038819875776397513,
"value": 51
},
{
"pct": 0.00019409937888198756,
"count": 1,
"cumFreq": 3,
"cumPct": 0.0005822981366459627,
"value": 49
},
...
]}}
With ZeppelinSolr the frequency table can be first visualized as a table:
The frequency table can then be plotted by switching to a scatter chart and selecting the value column for the xaxis and the count column for the yaxis
Notice that the visualization nicely displays the frequency of daily change in stock prices rounded to integers. The most frequently occurring value is 0 with 1494 occurrences followed by 1 and 1 with around 700 occurrences.
Percentiles
The percentile
function returns the estimated value for a specific percentile in
a sample set. The example below returns a random sample containing the response_d
field
from the logs collection. The response_d
field is vectorized and the 20th percentile
is calculated for the vector:
let(a=random(logs, q="*:*", rows="15000", fl="response_d"),
b=col(a, response_d),
c=percentile(b, 20))
When this expression is sent to the /stream
handler it responds with:
{
"resultset": {
"docs": [
{
"c": 818.073554
},
{
"EOF": true,
"RESPONSE_TIME": 286
}
]
}
}
The percentile
function can also compute an array of percentile values.
The example below is computing the 20th, 40th, 60th and 80th percentiles for a random sample
of the response_d
field:
let(a=random(logs, q="*:*", rows="15000", fl="response_d"),
b=col(a, response_d),
c=percentile(b, array(20,40,60,80)))
When this expression is sent to the /stream
handler it responds with:
{
"resultset": {
"docs": [
{
"c": [
818.0835543394625,
843.5590348165282,
866.1789509894824,
892.5033386599067
]
},
{
"EOF": true,
"RESPONSE_TIME": 291
}
]
}
}
Quantile Plots
Quantile plots or QQ Plots are powerful tools for visually comparing two or more distributions.
A quantile plot, plots the percentiles from two or more distributions in the same visualization. This allows for visual comparison of the distributions at each percentile. A simple example will help illustrate the power of quantile plots.
In this example the distribution of daily stock price changes for two stock tickers, goog and amzn, are visualized with a quantile plot.
The example first creates an array of values representing the percentiles that will be calculated and sets this array
to variable p. Then random samples of the change_d
field are drawn for the tickers amzn and goog. The change_d
field
represents the change in stock price for one day. Then the change_d
field is vectorized for both samples and placed
in the variables amzn and goog. The percentile
function is then used to calculate the percentiles for both vectors. Notice that
the variable p is used to specify the list of percentiles that are calculated.
Finally zplot
is used to plot the percentiles sequence on the xaxis and the calculated
percentile values for both distributions on the yaxis. And a line plot is used
to visualize the QQ plot.
This quantile plot provides a clear picture of the distributions of daily price changes for amzn and googl. In the plot the xaxis is the percentiles and the yaxis is the percentile value calculated.
Notice that the goog percentile value starts lower and ends higher than the amzn plot and that there is a steeper slope. This shows the greater variability in the goog price change distribution. The plot gives a clear picture of the difference in the distributions across the full range of percentiles.
Correlation and Covariance
Correlation and Covariance measure how random variables fluctuate together.
Correlation and Correlation Matrices
Correlation is a measure of the linear correlation between two vectors. Correlation is scaled between 1 and 1.
Three correlation types are supported:
pearsons (default)
kendalls
spearmans
The type of correlation is specified by adding the type named parameter in the function call.
In the example below a random sample containing two fields, filesize_d
and response_d
, is drawn from
the logs collection using the random
function. The fields are vectorized into the
variables x and y and then Spearman’s correlation for
the two vectors is calculated using the corr
function.
Correlation Matrices
Correlation matrices are powerful tools for visualizing the correlation between two or more vectors.
The corr
function builds a correlation matrix
if a matrix is passed as the parameter. The correlation matrix is computed by correlating the columns
of the matrix.
The example below demonstrates the power of correlation matrices combined with 2 dimensional faceting.
In this example the facet2D
function is used to generate a two dimensional facet aggregation
over the fields complaint_type_s
and zip_s
from the nyc311 complaints database.
The top 20 complaint types and the top 25 zip codes for each complaint type are aggregated.
The result is a stream of tuples each containing the fields complaint_type_s
, zip_s
and
the count for the pair.
The pivot
function is then used to pivot the fields into a matrix with the zip_s
field as the rows and the complaint_type_s
field as the columns. The count(*)
field populates
the values in the cells of the matrix.
The corr
function is then used correlate the columns of the matrix. This produces a correlation matrix
that shows how complaint types are correlated based on the zip codes they appear in. Another way to look at this
is it shows how the different complaint types tend to cooccur across zip codes.
Finally the zplot
function is used to plot the correlation matrix as a heat map.
Notice in the example the correlation matrix is square with complaint types shown on both the x and yaxises. The color of the cells in the heat map shows the intensity of the correlation between the complaint types.
The heat map is interactive, so mousing over one of the cells pops up the values for the cell.
Notice that HEAT/HOT WATER and UNSANITARY CONDITION complaints have a correlation of 8 (rounded to the nearest tenth).
Covariance and Covariance Matrices
Covariance is an unscaled measure of correlation.
The cov
function calculates the covariance of two vectors of data.
In the example below a random sample containing two fields, filesize_d
and response_d
, is drawn from
the logs collection using the random
function. The fields are vectorized into the
variables x and y and then the covariance for
the two vectors is calculated using the cov
function.
If a matrix is passed to the cov
function it will automatically compute a covariance
matrix for the columns of the matrix.
Notice in the example below that the x and y vectors are added to a matrix. The matrix is then transposed to turn the rows into columns, and the covariance matrix is computed for the columns of the matrix.
let(a=random(logs, q="*:*", fl="filesize_d, response_d", rows=50000),
x=col(a, filesize_d),
y=col(a, response_d),
m=transpose(matrix(x, y)),
covariance=cov(m))
When this expression is sent to the /stream
handler it responds with:
{
"resultset": {
"docs": [
{
"covariance": [
[
4018404.072532102,
80243.3948172242
],
[
80243.3948172242,
1948.3216661122592
]
]
},
{
"EOF": true,
"RESPONSE_TIME": 534
}
]
}
}
The covariance matrix contains both the variance for the two vectors and the covariance between the vectors in the following format:
x y
x [4018404.072532102, 80243.3948172242],
y [80243.3948172242, 1948.3216661122592]
The covariance matrix is always square. So a covariance matrix created from 3 vectors will produce a 3 x 3 matrix.
Statistical Inference Tests
Statistical inference tests test a hypothesis on random samples and return pvalues which can be used to infer the reliability of the test for the entire population.
The following statistical inference tests are available:
anova
: OneWayAnova tests if there is a statistically significant difference in the means of two or more random samples.ttest
: The Ttest tests if there is a statistically significant difference in the means of two random samples.pairedTtest
: The paired ttest tests if there is a statistically significant difference in the means of two random samples with paired data.gTestDataSet
: The Gtest tests if two samples of binned discrete data were drawn from the same population.chiSquareDataset
: The ChiSquared test tests if two samples of binned discrete data were drawn from the same population.mannWhitney
: The MannWhitney test is a nonparametric test that tests if two samples of continuous data were pulled from the same population. The MannWhitney test is often used instead of the Ttest when the underlying assumptions of the Ttest are not met.ks
: The KolmogorovSmirnov test tests if two samples of continuous data were drawn from the same distribution.
Below is a simple example of a Ttest performed on two random samples. The returned pvalue of .93 means we can accept the null hypothesis that the two samples do not have statistically significantly differences in the means.
let(a=random(collection1, q="*:*", rows="1500", fl="price_f"),
b=random(collection1, q="*:*", rows="1500", fl="price_f"),
c=col(a, price_f),
d=col(b, price_f),
e=ttest(c, d))
When this expression is sent to the /stream
handler it responds with:
{
"resultset": {
"docs": [
{
"e": {
"pvalue": 0.9350135639249795,
"tstatistic": 0.081545541074817
}
},
{
"EOF": true,
"RESPONSE_TIME": 48
}
]
}
}
Transformations
In statistical analysis its often useful to transform data sets before performing statistical calculations. The statistical function library includes the following commonly used transformations:
rank
: Returns a numeric array with the ranktransformed value of each element of the original array.log
: Returns a numeric array with the natural log of each element of the original array.log10
: Returns a numeric array with the base 10 log of each element of the original array.sqrt
: Returns a numeric array with the square root of each element of the original array.cbrt
: Returns a numeric array with the cube root of each element of the original array.recip
: Returns a numeric array with the reciprocal of each element of the original array.
Below is an example of a ttest performed on log transformed data sets:
let(a=random(collection1, q="*:*", rows="1500", fl="price_f"),
b=random(collection1, q="*:*", rows="1500", fl="price_f"),
c=log(col(a, price_f)),
d=log(col(b, price_f)),
e=ttest(c, d))
When this expression is sent to the /stream
handler it responds with:
{
"resultset": {
"docs": [
{
"e": {
"pvalue": 0.9655110070265056,
"tstatistic": 0.04324265449471238
}
},
{
"EOF": true,
"RESPONSE_TIME": 58
}
]
}
}
Back Transformations
Vectors that have been transformed with the log
, log10
, sqrt
and cbrt
functions
can be back transformed using the pow
function.
The example below shows how to back transform data that has been transformed by the
sqrt
function.
let(echo="b,c",
a=array(100, 200, 300),
b=sqrt(a),
c=pow(b, 2))
When this expression is sent to the /stream
handler it responds with:
{
"resultset": {
"docs": [
{
"b": [
10,
14.142135623730951,
17.320508075688775
],
"c": [
100,
200.00000000000003,
300.00000000000006
]
},
{
"EOF": true,
"RESPONSE_TIME": 0
}
]
}
}
The example below shows how to back transform data that has been transformed by the
log10
function.
let(echo="b,c",
a=array(100, 200, 300),
b=log10(a),
c=pow(10, b))
When this expression is sent to the /stream
handler it responds with:
{
"resultset": {
"docs": [
{
"b": [
2,
2.3010299956639813,
2.4771212547196626
],
"c": [
100,
200.00000000000003,
300.0000000000001
]
},
{
"EOF": true,
"RESPONSE_TIME": 0
}
]
}
}
Vectors that have been transformed with the recip
function can be backtransformed by taking the reciprocal
of the reciprocal.
The example below shows an example of the backtransformation of the recip
function.
let(echo="b,c",
a=array(100, 200, 300),
b=recip(a),
c=recip(b))
When this expression is sent to the /stream
handler it responds with:
{
"resultset": {
"docs": [
{
"b": [
0.01,
0.005,
0.0033333333333333335
],
"c": [
100,
200,
300
]
},
{
"EOF": true,
"RESPONSE_TIME": 0
}
]
}
}
Zscores
The zscores
function converts a numeric array to an array of zscores. The zscore
is the number of standard deviations a number is from the mean.
The example below computes the zscores for the values in an array.
let(a=array(1,2,3),
b=zscores(a))
When this expression is sent to the /stream
handler it responds with:
{
"resultset": {
"docs": [
{
"b": [
1,
0,
1
]
},
{
"EOF": true,
"RESPONSE_TIME": 27
}
]
}
}