Package org.HdrHistogram

Class AbstractHistogram

  • All Implemented Interfaces:
    Serializable, ValueRecorder
    Direct Known Subclasses:
    Histogram, IntCountsHistogram, ShortCountsHistogram
    public abstract class AbstractHistogram
extends EncodableHistogram
implements ValueRecorder, Serializable

    An abstract base class for integer values High Dynamic Range (HDR) Histograms

    AbstractHistogram supports the recording and analyzing sampled data value counts across a configurable integer value range with configurable value precision within the range. Value precision is expressed as the number of significant digits in the value recording, and provides control over value quantization behavior across the value range and the subsequent value resolution at any given level.

    For example, a Histogram could be configured to track the counts of observed integer values between 0 and 3,600,000,000 while maintaining a value precision of 3 significant digits across that range. Value quantization within the range will thus be no larger than 1/1,000th (or 0.1%) of any value. This example Histogram could be used to track and analyze the counts of observed response times ranging between 1 microsecond and 1 hour in magnitude, while maintaining a value resolution of 1 microsecond up to 1 millisecond, a resolution of 1 millisecond (or better) up to one second, and a resolution of 1 second (or better) up to 1,000 seconds. At it's maximum tracked value (1 hour), it would still maintain a resolution of 3.6 seconds (or better).

    See package description for org.HdrHistogram for details.

    See Also:
    Serialized Form

    • Constructor Detail

      • AbstractHistogram

        protected AbstractHistogram​(int numberOfSignificantValueDigits)
        Construct an auto-resizing histogram with a lowest discernible value of 1 and an auto-adjusting highestTrackableValue. Can auto-resize up to track values up to (Long.MAX_VALUE / 2).
        Parameters:
        numberOfSignificantValueDigits - The number of significant decimal digits to which the histogram will maintain value resolution and separation. Must be a non-negative integer between 0 and 5.

      • AbstractHistogram

        protected AbstractHistogram​(long lowestDiscernibleValue,
                            long highestTrackableValue,
                            int numberOfSignificantValueDigits)
        Construct a histogram given the Lowest and Highest values to be tracked and a number of significant decimal digits. Providing a lowestDiscernibleValue is useful is situations where the units used for the histogram's values are much smaller that the minimal accuracy required. E.g. when tracking time values stated in nanosecond units, where the minimal accuracy required is a microsecond, the proper value for lowestDiscernibleValue would be 1000.
        Parameters:
        lowestDiscernibleValue - The lowest value that can be discerned (distinguished from 0) by the histogram. Must be a positive integer that is >= 1. May be internally rounded down to nearest power of 2.
        highestTrackableValue - The highest value to be tracked by the histogram. Must be a positive integer that is >= (2 * lowestDiscernibleValue).
        numberOfSignificantValueDigits - The number of significant decimal digits to which the histogram will maintain value resolution and separation. Must be a non-negative integer between 0 and 5.

      • AbstractHistogram

        protected AbstractHistogram​(AbstractHistogram source)
        Construct a histogram with the same range settings as a given source histogram, duplicating the source's start/end timestamps (but NOT it's contents)
        Parameters:
        source - The source histogram to duplicate

    • Method Detail

      • getTotalCount

        public abstract long getTotalCount()
        Get the total count of all recorded values in the histogram
        Returns:
        the total count of all recorded values in the histogram

      • isAutoResize

        public boolean isAutoResize()
        Indicate whether or not the histogram is set to auto-resize and auto-adjust it's highestTrackableValue
        Returns:
        autoResize setting

      • supportsAutoResize

        public boolean supportsAutoResize()
        Indicate whether or not the histogram is capable of supporting auto-resize functionality. Note that this is an indication that enabling auto-resize by calling setAutoResize() is allowed, and NOT that the histogram will actually auto-resize. Use isAutoResize() to determine if the histogram is in auto-resize mode.
        Returns:
        autoResize setting

      • setAutoResize

        public void setAutoResize​(boolean autoResize)
        Control whether or not the histogram can auto-resize and auto-adjust it's highestTrackableValue
        Parameters:
        autoResize - autoResize setting

      • recordValueWithExpectedInterval

        public void recordValueWithExpectedInterval​(long value,
                                            long expectedIntervalBetweenValueSamples)
                                     throws ArrayIndexOutOfBoundsException
        Record a value in the histogram.

        To compensate for the loss of sampled values when a recorded value is larger than the expected interval between value samples, Histogram will auto-generate an additional series of decreasingly-smaller (down to the expectedIntervalBetweenValueSamples) value records.

        Note: This is a at-recording correction method, as opposed to the post-recording correction method provided by copyCorrectedForCoordinatedOmission(long). The two methods are mutually exclusive, and only one of the two should be be used on a given data set to correct for the same coordinated omission issue.

        See notes in the description of the Histogram calls for an illustration of why this corrective behavior is important.

        Specified by:
        recordValueWithExpectedInterval in interface ValueRecorder
        Parameters:
        value - The value to record
        expectedIntervalBetweenValueSamples - If expectedIntervalBetweenValueSamples is larger than 0, add auto-generated value records as appropriate if value is larger than expectedIntervalBetweenValueSamples
        Throws:
        ArrayIndexOutOfBoundsException - (may throw) if value is exceeds highestTrackableValue

      • recordValue

        public void recordValue​(long value,
                        long expectedIntervalBetweenValueSamples)
                 throws ArrayIndexOutOfBoundsException
        Deprecated.
        Record a value in the histogram. This deprecated method has identical behavior to recordValueWithExpectedInterval(). It was renamed to avoid ambiguity.
        Parameters:
        value - The value to record
        expectedIntervalBetweenValueSamples - If expectedIntervalBetweenValueSamples is larger than 0, add auto-generated value records as appropriate if value is larger than expectedIntervalBetweenValueSamples
        Throws:
        ArrayIndexOutOfBoundsException - (may throw) if value is exceeds highestTrackableValue

      • reset

        public void reset()
        Reset the contents and stats of this histogram
        Specified by:
        reset in interface ValueRecorder

      • copy

        public abstract AbstractHistogram copy()
        Create a copy of this histogram, complete with data and everything.
        Returns:
        A distinct copy of this histogram.

      • copyCorrectedForCoordinatedOmission

        public abstract AbstractHistogram copyCorrectedForCoordinatedOmission​(long expectedIntervalBetweenValueSamples)
        Get a copy of this histogram, corrected for coordinated omission.

        To compensate for the loss of sampled values when a recorded value is larger than the expected interval between value samples, the new histogram will include an auto-generated additional series of decreasingly-smaller (down to the expectedIntervalBetweenValueSamples) value records for each count found in the current histogram that is larger than the expectedIntervalBetweenValueSamples. Note: This is a post-correction method, as opposed to the at-recording correction method provided by recordValueWithExpectedInterval. The two methods are mutually exclusive, and only one of the two should be be used on a given data set to correct for the same coordinated omission issue. by

        See notes in the description of the Histogram calls for an illustration of why this corrective behavior is important.

        Parameters:
        expectedIntervalBetweenValueSamples - If expectedIntervalBetweenValueSamples is larger than 0, add auto-generated value records as appropriate if value is larger than expectedIntervalBetweenValueSamples
        Returns:
        a copy of this histogram, corrected for coordinated omission.

      • copyInto

        public void copyInto​(AbstractHistogram targetHistogram)
        Copy this histogram into the target histogram, overwriting it's contents.
        Parameters:
        targetHistogram - the histogram to copy into

      • copyIntoCorrectedForCoordinatedOmission

        public void copyIntoCorrectedForCoordinatedOmission​(AbstractHistogram targetHistogram,
                                                    long expectedIntervalBetweenValueSamples)
        Copy this histogram, corrected for coordinated omission, into the target histogram, overwriting it's contents. (see copyCorrectedForCoordinatedOmission(long) for more detailed explanation about how correction is applied)
        Parameters:
        targetHistogram - the histogram to copy into
        expectedIntervalBetweenValueSamples - If expectedIntervalBetweenValueSamples is larger than 0, add auto-generated value records as appropriate if value is larger than expectedIntervalBetweenValueSamples

      • add

        public void add​(AbstractHistogram otherHistogram)
         throws ArrayIndexOutOfBoundsException
        Add the contents of another histogram to this one.

        As part of adding the contents, the start/end timestamp range of this histogram will be extended to include the start/end timestamp range of the other histogram.

        Parameters:
        otherHistogram - The other histogram.
        Throws:
        ArrayIndexOutOfBoundsException - (may throw) if values in fromHistogram's are higher than highestTrackableValue.

      • addWhileCorrectingForCoordinatedOmission

        public void addWhileCorrectingForCoordinatedOmission​(AbstractHistogram otherHistogram,
                                                     long expectedIntervalBetweenValueSamples)
        Add the contents of another histogram to this one, while correcting the incoming data for coordinated omission.

        To compensate for the loss of sampled values when a recorded value is larger than the expected interval between value samples, the values added will include an auto-generated additional series of decreasingly-smaller (down to the expectedIntervalBetweenValueSamples) value records for each count found in the current histogram that is larger than the expectedIntervalBetweenValueSamples. Note: This is a post-recording correction method, as opposed to the at-recording correction method provided by recordValueWithExpectedInterval. The two methods are mutually exclusive, and only one of the two should be be used on a given data set to correct for the same coordinated omission issue. by

        See notes in the description of the Histogram calls for an illustration of why this corrective behavior is important.

        Parameters:
        otherHistogram - The other histogram. highestTrackableValue and largestValueWithSingleUnitResolution must match.
        expectedIntervalBetweenValueSamples - If expectedIntervalBetweenValueSamples is larger than 0, add auto-generated value records as appropriate if value is larger than expectedIntervalBetweenValueSamples
        Throws:
        ArrayIndexOutOfBoundsException - (may throw) if values exceed highestTrackableValue

      • shiftValuesLeft

        public void shiftValuesLeft​(int numberOfBinaryOrdersOfMagnitude)
        Shift recorded values to the left (the equivalent of a << shift operation on all recorded values). The configured integer value range limits and value precision setting will remain unchanged. An ArrayIndexOutOfBoundsException will be thrown if any recorded values may be lost as a result of the attempted operation, reflecting an "overflow" conditions. Expect such an overflow exception if the operation would cause the current maxValue to be scaled to a value that is outside of the covered value range.
        Parameters:
        numberOfBinaryOrdersOfMagnitude - The number of binary orders of magnitude to shift by

      • shiftValuesRight

        public void shiftValuesRight​(int numberOfBinaryOrdersOfMagnitude)
        Shift recorded values to the right (the equivalent of a >> shift operation on all recorded values). The configured integer value range limits and value precision setting will remain unchanged.

        Shift right operations that do not underflow are reversible with a shift left operation with no loss of information. An ArrayIndexOutOfBoundsException reflecting an "underflow" conditions will be thrown if any recorded values may lose representation accuracy as a result of the attempted shift operation.

        For a shift of a single order of magnitude, expect such an underflow exception if any recorded non-zero values up to [numberOfSignificantValueDigits (rounded up to nearest power of 2) multiplied by (2 ^ numberOfBinaryOrdersOfMagnitude) currently exist in the histogram.

        Parameters:
        numberOfBinaryOrdersOfMagnitude - The number of binary orders of magnitude to shift by

      • equals

        public boolean equals​(Object other)
        Determine if this histogram is equivalent to another.
        Overrides:
        equals in class Object
        Parameters:
        other - the other histogram to compare to
        Returns:
        True if this histogram are equivalent with the other.

      • hashCode

        public int hashCode()
        Overrides:
        hashCode in class Object

      • getLowestDiscernibleValue

        public long getLowestDiscernibleValue()
        get the configured lowestDiscernibleValue
        Returns:
        lowestDiscernibleValue

      • getHighestTrackableValue

        public long getHighestTrackableValue()
        get the configured highestTrackableValue
        Returns:
        highestTrackableValue

      • getNumberOfSignificantValueDigits

        public int getNumberOfSignificantValueDigits()
        get the configured numberOfSignificantValueDigits
        Returns:
        numberOfSignificantValueDigits

      • sizeOfEquivalentValueRange

        public long sizeOfEquivalentValueRange​(long value)
        Get the size (in value units) of the range of values that are equivalent to the given value within the histogram's resolution. Where "equivalent" means that value samples recorded for any two equivalent values are counted in a common total count.
        Parameters:
        value - The given value
        Returns:
        The size of the range of values equivalent to the given value.

      • lowestEquivalentValue

        public long lowestEquivalentValue​(long value)
        Get the lowest value that is equivalent to the given value within the histogram's resolution. Where "equivalent" means that value samples recorded for any two equivalent values are counted in a common total count.
        Parameters:
        value - The given value
        Returns:
        The lowest value that is equivalent to the given value within the histogram's resolution.

      • highestEquivalentValue

        public long highestEquivalentValue​(long value)
        Get the highest value that is equivalent to the given value within the histogram's resolution. Where "equivalent" means that value samples recorded for any two equivalent values are counted in a common total count.
        Parameters:
        value - The given value
        Returns:
        The highest value that is equivalent to the given value within the histogram's resolution.

      • medianEquivalentValue

        public long medianEquivalentValue​(long value)
        Get a value that lies in the middle (rounded up) of the range of values equivalent the given value. Where "equivalent" means that value samples recorded for any two equivalent values are counted in a common total count.
        Parameters:
        value - The given value
        Returns:
        The value lies in the middle (rounded up) of the range of values equivalent the given value.

      • nextNonEquivalentValue

        public long nextNonEquivalentValue​(long value)
        Get the next value that is not equivalent to the given value within the histogram's resolution. Where "equivalent" means that value samples recorded for any two equivalent values are counted in a common total count.
        Parameters:
        value - The given value
        Returns:
        The next value that is not equivalent to the given value within the histogram's resolution.

      • valuesAreEquivalent

        public boolean valuesAreEquivalent​(long value1,
                                   long value2)
        Determine if two values are equivalent with the histogram's resolution. Where "equivalent" means that value samples recorded for any two equivalent values are counted in a common total count.
        Parameters:
        value1 - first value to compare
        value2 - second value to compare
        Returns:
        True if values are equivalent with the histogram's resolution.

      • getEstimatedFootprintInBytes

        public int getEstimatedFootprintInBytes()
        Provide a (conservatively high) estimate of the Histogram's total footprint in bytes
        Returns:
        a (conservatively high) estimate of the Histogram's total footprint in bytes

      • getStartTimeStamp

        public long getStartTimeStamp()
        get the start time stamp [optionally] stored with this histogram
        Specified by:
        getStartTimeStamp in class EncodableHistogram
        Returns:
        the start time stamp [optionally] stored with this histogram

      • setStartTimeStamp

        public void setStartTimeStamp​(long timeStampMsec)
        Set the start time stamp value associated with this histogram to a given value.
        Specified by:
        setStartTimeStamp in class EncodableHistogram
        Parameters:
        timeStampMsec - the value to set the time stamp to, [by convention] in msec since the epoch.

      • getEndTimeStamp

        public long getEndTimeStamp()
        get the end time stamp [optionally] stored with this histogram
        Specified by:
        getEndTimeStamp in class EncodableHistogram
        Returns:
        the end time stamp [optionally] stored with this histogram

      • setEndTimeStamp

        public void setEndTimeStamp​(long timeStampMsec)
        Set the end time stamp value associated with this histogram to a given value.
        Specified by:
        setEndTimeStamp in class EncodableHistogram
        Parameters:
        timeStampMsec - the value to set the time stamp to, [by convention] in msec since the epoch.

      • getTag

        public String getTag()
        get the tag string [optionally] associated with this histogram
        Specified by:
        getTag in class EncodableHistogram
        Returns:
        tag string [optionally] associated with this histogram

      • setTag

        public void setTag​(String tag)
        Set the tag string associated with this histogram
        Specified by:
        setTag in class EncodableHistogram
        Parameters:
        tag - the tag string to associate with this histogram

      • getMinValue

        public long getMinValue()
        Get the lowest recorded value level in the histogram. If the histogram has no recorded values, the value returned is undefined.
        Returns:
        the Min value recorded in the histogram

      • getMaxValue

        public long getMaxValue()
        Get the highest recorded value level in the histogram. If the histogram has no recorded values, the value returned is undefined.
        Returns:
        the Max value recorded in the histogram

      • getMinNonZeroValue

        public long getMinNonZeroValue()
        Get the lowest recorded non-zero value level in the histogram. If the histogram has no recorded values, the value returned is undefined.
        Returns:
        the lowest recorded non-zero value level in the histogram

      • getMaxValueAsDouble

        public double getMaxValueAsDouble()
        Get the highest recorded value level in the histogram as a double
        Specified by:
        getMaxValueAsDouble in class EncodableHistogram
        Returns:
        the Max value recorded in the histogram

      • getMean

        public double getMean()
        Get the computed mean value of all recorded values in the histogram
        Returns:
        the mean value (in value units) of the histogram data

      • getStdDeviation

        public double getStdDeviation()
        Get the computed standard deviation of all recorded values in the histogram
        Returns:
        the standard deviation (in value units) of the histogram data

      • getValueAtPercentile

        public long getValueAtPercentile​(double percentile)
        Get the value at a given percentile. Returns the largest value that (100% - percentile) [+/- 1 ulp] of the overall recorded value entries in the histogram are either larger than or equivalent to. Returns 0 if no recorded values exist.

        Note that two values are "equivalent" in this statement if valuesAreEquivalent(long, long) would return true.

        Parameters:
        percentile - The percentile for which to return the associated value
        Returns:
        The largest value that (100% - percentile) [+/- 1 ulp] of the overall recorded value entries in the histogram are either larger than or equivalent to. Returns 0 if no recorded values exist.

      • getPercentileAtOrBelowValue

        public double getPercentileAtOrBelowValue​(long value)
        Get the percentile at a given value. The percentile returned is the percentile of values recorded in the histogram that are smaller than or equivalent to the given value.

        Note that two values are "equivalent" in this statement if valuesAreEquivalent(long, long) would return true.

        Parameters:
        value - The value for which to return the associated percentile
        Returns:
        The percentile of values recorded in the histogram that are smaller than or equivalent to the given value.

      • getCountBetweenValues

        public long getCountBetweenValues​(long lowValue,
                                  long highValue)
                           throws ArrayIndexOutOfBoundsException
        Get the count of recorded values within a range of value levels (inclusive to within the histogram's resolution).
        Parameters:
        lowValue - The lower value bound on the range for which to provide the recorded count. Will be rounded down with lowestEquivalentValue.
        highValue - The higher value bound on the range for which to provide the recorded count. Will be rounded up with highestEquivalentValue.
        Returns:
        the total count of values recorded in the histogram within the value range that is >= lowestEquivalentValue(lowValue) and <= highestEquivalentValue(highValue)
        Throws:
        ArrayIndexOutOfBoundsException

      • getCountAtValue

        public long getCountAtValue​(long value)
                     throws ArrayIndexOutOfBoundsException
        Get the count of recorded values at a specific value (to within the histogram resolution at the value level).
        Parameters:
        value - The value for which to provide the recorded count
        Returns:
        The total count of values recorded in the histogram within the value range that is >= lowestEquivalentValue(value) and <= highestEquivalentValue(value)
        Throws:
        ArrayIndexOutOfBoundsException

      • percentiles

        public AbstractHistogram.Percentiles percentiles​(int percentileTicksPerHalfDistance)
        Provide a means of iterating through histogram values according to percentile levels. The iteration is performed in steps that start at 0% and reduce their distance to 100% according to the percentileTicksPerHalfDistance parameter, ultimately reaching 100% when all recorded histogram values are exhausted.
        Parameters:
        percentileTicksPerHalfDistance - The number of iteration steps per half-distance to 100%.
        Returns:
        An Iterable<HistogramIterationValue> through the histogram using a PercentileIterator

      • linearBucketValues

        public AbstractHistogram.LinearBucketValues linearBucketValues​(long valueUnitsPerBucket)
        Provide a means of iterating through histogram values using linear steps. The iteration is performed in steps of valueUnitsPerBucket in size, terminating when all recorded histogram values are exhausted.
        Parameters:
        valueUnitsPerBucket - The size (in value units) of the linear buckets to use
        Returns:
        An Iterable<HistogramIterationValue> through the histogram using a LinearIterator

      • logarithmicBucketValues

        public AbstractHistogram.LogarithmicBucketValues logarithmicBucketValues​(long valueUnitsInFirstBucket,
                                                                         double logBase)
        Provide a means of iterating through histogram values at logarithmically increasing levels. The iteration is performed in steps that start at valueUnitsInFirstBucket and increase exponentially according to logBase, terminating when all recorded histogram values are exhausted.
        Parameters:
        valueUnitsInFirstBucket - The size (in value units) of the first bucket in the iteration
        logBase - The multiplier by which bucket sizes will grow in each iteration step
        Returns:
        An Iterable<HistogramIterationValue> through the histogram using a LogarithmicIterator

      • recordedValues

        public AbstractHistogram.RecordedValues recordedValues()
        Provide a means of iterating through all recorded histogram values using the finest granularity steps supported by the underlying representation. The iteration steps through all non-zero recorded value counts, and terminates when all recorded histogram values are exhausted.
        Returns:
        An Iterable<HistogramIterationValue> through the histogram using a RecordedValuesIterator

      • allValues

        public AbstractHistogram.AllValues allValues()
        Provide a means of iterating through all histogram values using the finest granularity steps supported by the underlying representation. The iteration steps through all possible unit value levels, regardless of whether or not there were recorded values for that value level, and terminates when all recorded histogram values are exhausted.
        Returns:
        An Iterable<HistogramIterationValue> through the histogram using a AllValuesIterator

      • outputPercentileDistribution

        public void outputPercentileDistribution​(PrintStream printStream,
                                         Double outputValueUnitScalingRatio)
        Produce textual representation of the value distribution of histogram data by percentile. The distribution is output with exponentially increasing resolution, with each exponentially decreasing half-distance containing five (5) percentile reporting tick points.
        Parameters:
        printStream - Stream into which the distribution will be output
        outputValueUnitScalingRatio - The scaling factor by which to divide histogram recorded values units in output

      • outputPercentileDistribution

        public void outputPercentileDistribution​(PrintStream printStream,
                                         int percentileTicksPerHalfDistance,
                                         Double outputValueUnitScalingRatio)
        Produce textual representation of the value distribution of histogram data by percentile. The distribution is output with exponentially increasing resolution, with each exponentially decreasing half-distance containing dumpTicksPerHalf percentile reporting tick points.
        Parameters:
        printStream - Stream into which the distribution will be output
        percentileTicksPerHalfDistance - The number of reporting points per exponentially decreasing half-distance
        outputValueUnitScalingRatio - The scaling factor by which to divide histogram recorded values units in output

      • outputPercentileDistribution

        public void outputPercentileDistribution​(PrintStream printStream,
                                         int percentileTicksPerHalfDistance,
                                         Double outputValueUnitScalingRatio,
                                         boolean useCsvFormat)
        Produce textual representation of the value distribution of histogram data by percentile. The distribution is output with exponentially increasing resolution, with each exponentially decreasing half-distance containing dumpTicksPerHalf percentile reporting tick points.
        Parameters:
        printStream - Stream into which the distribution will be output
        percentileTicksPerHalfDistance - The number of reporting points per exponentially decreasing half-distance
        outputValueUnitScalingRatio - The scaling factor by which to divide histogram recorded values units in output
        useCsvFormat - Output in CSV format if true. Otherwise use plain text form.

      • getNeededByteBufferCapacity

        public int getNeededByteBufferCapacity()
        Get the capacity needed to encode this histogram into a ByteBuffer
        Specified by:
        getNeededByteBufferCapacity in class EncodableHistogram
        Returns:
        the capacity needed to encode this histogram into a ByteBuffer

      • encodeIntoByteBuffer

        public int encodeIntoByteBuffer​(ByteBuffer buffer)
        Encode this histogram into a ByteBuffer
        Parameters:
        buffer - The buffer to encode into
        Returns:
        The number of bytes written to the buffer

      • encodeIntoCompressedByteBuffer

        public int encodeIntoCompressedByteBuffer​(ByteBuffer targetBuffer,
                                          int compressionLevel)
        Encode this histogram in compressed form into a byte array
        Specified by:
        encodeIntoCompressedByteBuffer in class EncodableHistogram
        Parameters:
        targetBuffer - The buffer to encode into
        compressionLevel - Compression level (for java.util.zip.Deflater).
        Returns:
        The number of bytes written to the buffer

      • encodeIntoCompressedByteBuffer

        public int encodeIntoCompressedByteBuffer​(ByteBuffer targetBuffer)
        Encode this histogram in compressed form into a byte array
        Parameters:
        targetBuffer - The buffer to encode into
        Returns:
        The number of bytes written to the array