Statistics

histograms

In the example below a histogram is used to visualize a random sample of response times from the logs collection. The example retrieves the random sample with the random function and creates a vector from the response_d field in the result set. Then the hist function is applied to the vector to return a histogram with 22 bins. The hist function returns a list of tuples with summary statistics for each bin.

let(a=random(logs, q="*:*", fl="response_d", rows="50000"),
    b=col(a, response_d),
    c=hist(b,  22))

When this expression is sent to the /stream handler it responds with:

{
  "result-set": {
    "docs": [
      {
        "prob": 0.00004896007228311655,
        "min": 675.573084576817,
        "max": 688.3309631697003,
        "mean": 683.805542728906,
        "var": 50.9974629924082,
        "cumProb": 0.000030022417162809913,
        "sum": 2051.416628186718,
        "stdev": 7.141250800273591,
        "N": 3
      },
      {
        "prob": 0.00029607514624062624,
        "min": 696.2875238591652,
        "max": 707.9706315779541,
        "mean": 702.1110569558929,
        "var": 14.136444379466969,
        "cumProb": 0.00022705264963879807,
        "sum": 11233.776911294284,
        "stdev": 3.759846323916307,
        "N": 16
      },
      {
        "prob": 0.0011491235433157194,
        "min": 709.1574910598678,
        "max": 724.9027194369135,
        "mean": 717.8554290699951,
        "var": 20.6935845290122,
        "cumProb": 0.0009858515418689757,
        "sum": 41635.61488605971,
        "stdev": 4.549020172412098,
        "N": 58
      },
      ...
      ]}}

With Zeppelin-Solr the histogram can be first visualized as a table:

Then the histogram can be visualized with an area chart by plotting the mean of the bins on the x-axis and the prob (probability) on the y-axis:

The cumulative probability can be plotted by switching the y-axis to the cumProb column:

Custom Histograms

Custom histograms can be defined and visualized by combining the output from multiple stats functions into a single histogram. Instead of automatically binning a numeric field the custom histogram allows for comparison of bins based on queries.

A simple example will illustrate how to define and visualize a custom histogram.

In this example, three stats functions are wrapped in a plist function. The plist (parallel list) function executes each of its internal functions in parallel and concatenates the results into a single stream. plist also maintains the order of the outputs from each of the sub-functions. In this example each stats function computes the count of documents that match a specific query. In this case they count the number of documents that contain the terms copper, gold and silver. The list of tuples with the counts is then stored in variable a.

Then an array of labels is created and set to variable l.

Finally the zplot function is used to plot the labels vector and the count(*) column. Notice the col function is used inside of the zplot function to extract the counts from the stats results.

Frequency Tables

The freqTable function returns a frequency distribution for a discrete data set. The freqTable function doesn’t create bins like the histogram. Instead it counts the occurrence of each discrete data value and returns a list of tuples with the frequency statistics for each value.

Below is an example of a frequency table built from a result set of rounded differences in daily opening stock prices for the stock ticker amzn.

This example is interesting because it shows a multi-step process to arrive at the result. The first step is to search for records in the the stocks collection with a ticker of amzn. Notice that the result set is sorted by date ascending and it returns the open_d field which is the opening price for the day.

The open_d field is then vectorized and set to variable b, which now contains a vector of opening prices ordered by date ascending.

The diff function is then used to calculate the first difference for the vector of opening prices. The first difference simply subtracts the previous value from each value in the array. This will provide an array of price differences for each day which will show daily change in opening price.

Then the round function is used to round the price differences to the nearest integer to create a vector of discrete values. The round function in this example is effectively binning continuous data at integer boundaries.

Finally the freqTable function is run on the discrete values to calculate the frequency table.

let(a=search(stocks,
             q="ticker_s:amzn",
             fl="open_d, date_dt",
             sort="date_dt asc",
             rows=25000),
    b=col(a, open_d),
    c=diff(b),
    d=round(c),
    e=freqTable(d))

When this expression is sent to the /stream handler it responds with:

 {
   "result-set": {
     "docs": [
       {
         "pct": 0.00019409937888198756,
         "count": 1,
         "cumFreq": 1,
         "cumPct": 0.00019409937888198756,
         "value": -57
       },
       {
         "pct": 0.00019409937888198756,
         "count": 1,
         "cumFreq": 2,
         "cumPct": 0.00038819875776397513,
         "value": -51
       },
       {
         "pct": 0.00019409937888198756,
         "count": 1,
         "cumFreq": 3,
         "cumPct": 0.0005822981366459627,
         "value": -49
       },
       ...
       ]}}

With Zeppelin-Solr the frequency table can be first visualized as a table:

The frequency table can then be plotted by switching to a scatter chart and selecting the value column for the x-axis and the count column for the y-axis

Notice that the visualization nicely displays the frequency of daily change in stock prices rounded to integers. The most frequently occurring value is 0 with 1494 occurrences followed by -1 and 1 with around 700 occurrences.

Quantile Plots

Quantile plots or QQ Plots are powerful tools for visually comparing two or more distributions.

A quantile plot, plots the percentiles from two or more distributions in the same visualization. This allows for visual comparison of the distributions at each percentile. A simple example will help illustrate the power of quantile plots.

In this example the distribution of daily stock price changes for two stock tickers, goog and amzn, are visualized with a quantile plot.

The example first creates an array of values representing the percentiles that will be calculated and sets this array to variable p. Then random samples of the change_d field are drawn for the tickers amzn and goog. The change_d field represents the change in stock price for one day. Then the change_d field is vectorized for both samples and placed in the variables amzn and goog. The percentile function is then used to calculate the percentiles for both vectors. Notice that the variable p is used to specify the list of percentiles that are calculated.

Finally zplot is used to plot the percentiles sequence on the x-axis and the calculated percentile values for both distributions on the y-axis. And a line plot is used to visualize the QQ plot.

This quantile plot provides a clear picture of the distributions of daily price changes for amzn and googl. In the plot the x-axis is the percentiles and the y-axis is the percentile value calculated.

Notice that the goog percentile value starts lower and ends higher than the amzn plot and that there is a steeper slope. This shows the greater variability in the goog price change distribution. The plot gives a clear picture of the difference in the distributions across the full range of percentiles.

Correlation and Correlation Matrices

Correlation is a measure of the linear correlation between two vectors. Correlation is scaled between -1 and 1.

Three correlation types are supported:

The type of correlation is specified by adding the type named parameter in the function call.

In the example below a random sample containing two fields, filesize_d and response_d, is drawn from the logs collection using the random function. The fields are vectorized into the variables x and y and then Spearman’s correlation for the two vectors is calculated using the corr function.

Correlation Matrices

Correlation matrices are powerful tools for visualizing the correlation between two or more vectors.

The corr function builds a correlation matrix if a matrix is passed as the parameter. The correlation matrix is computed by correlating the columns of the matrix.

The example below demonstrates the power of correlation matrices combined with 2 dimensional faceting.

In this example the facet2D function is used to generate a two dimensional facet aggregation over the fields complaint_type_s and zip_s from the nyc311 complaints database. The top 20 complaint types and the top 25 zip codes for each complaint type are aggregated. The result is a stream of tuples each containing the fields complaint_type_s, zip_s and the count for the pair.

The pivot function is then used to pivot the fields into a matrix with the zip_s field as the rows and the complaint_type_s field as the columns. The count(*) field populates the values in the cells of the matrix.

The corr function is then used correlate the columns of the matrix. This produces a correlation matrix that shows how complaint types are correlated based on the zip codes they appear in. Another way to look at this is it shows how the different complaint types tend to co-occur across zip codes.

Finally the zplot function is used to plot the correlation matrix as a heat map.

Notice in the example the correlation matrix is square with complaint types shown on both the x and y-axises. The color of the cells in the heat map shows the intensity of the correlation between the complaint types.

The heat map is interactive, so mousing over one of the cells pops up the values for the cell.

Notice that HEAT/HOT WATER and UNSANITARY CONDITION complaints have a correlation of 8 (rounded to the nearest tenth).

Covariance and Covariance Matrices

Covariance is an unscaled measure of correlation.

The cov function calculates the covariance of two vectors of data.

In the example below a random sample containing two fields, filesize_d and response_d, is drawn from the logs collection using the random function. The fields are vectorized into the variables x and y and then the covariance for the two vectors is calculated using the cov function.

If a matrix is passed to the cov function it will automatically compute a covariance matrix for the columns of the matrix.

Notice in the example below that the x and y vectors are added to a matrix. The matrix is then transposed to turn the rows into columns, and the covariance matrix is computed for the columns of the matrix.

let(a=random(logs, q="*:*", fl="filesize_d, response_d", rows=50000),
    x=col(a, filesize_d),
    y=col(a, response_d),
    m=transpose(matrix(x, y)),
    covariance=cov(m))

When this expression is sent to the /stream handler it responds with:

 {
   "result-set": {
     "docs": [
       {
         "covariance": [
           [
             4018404.072532102,
             80243.3948172242
           ],
           [
             80243.3948172242,
             1948.3216661122592
           ]
         ]
       },
       {
         "EOF": true,
         "RESPONSE_TIME": 534
       }
     ]
   }
 }

The covariance matrix contains both the variance for the two vectors and the covariance between the vectors in the following format:

         x                 y
 x [4018404.072532102, 80243.3948172242],
 y [80243.3948172242,  1948.3216661122592]

The covariance matrix is always square. So a covariance matrix created from 3 vectors will produce a 3 x 3 matrix.