Package io.engineblock.activityapi.ratelimits

Class HybridRateLimiter

  • All Implemented Interfaces:
    Startable, RateLimiter
    public class HybridRateLimiter
extends java.lang.Object
implements Startable, RateLimiter

    Synopsis

    This rate limiter uses nanoseconds as the unit of timing. This works well because it is the native precision of the system timer interface via System.nanoTime(). It is also low-error in terms of rounding between floating point rates and nanoseconds, at least in the round numbers that users tend to use. Further, the current scheduling state is maintained as an atomic view of accumulated nanoseconds granted to callers -- referred to here as the ticks accumulator. This further simplifies the implementation by allowing direct comparison of scheduled times with the current state of the high-resolution system timer.

    Design Notes

    This implementation makes certain trade-offs needed to support a combination of requirements. Specifically, some small degree of inaccuracy is allowed to enable higher throughput when needed. Some practical limitations affect how accurate we can be:
    1. This is not a real-time system with guarantees on scheduling.
    2. Calling overhead is significant for reading the RTC or sleeping.
    3. Controlling the accuracy of a delay is not possible under any level of load.
    4. It is undesirable (wasteful) to use spin loops to delay.
    Together, these factors mean a compromise is inevitable. In practice it means that a very accurate implementation will likely be very slow, and a very fast implementation will likely be very inaccurate. This implementation tries to strike a balance, providing accuracy near the microsecond level, while allowing rates in the tens of millions per second, even under heavy thread contention.

    Burst Ratio

    This rate limiter provides a sliding scale between strict rate limiting and average rate limiting, the difference between the two controlled by a burst ratio parameter. When the burst ratio is 1.0, the rate limiter acts as a strict rate limiter, disallowing faster operations from using time that was previously forfeited by prior slower operations. This is a "use it or lose it" mode that means things like GC events can steal throughput from a running client as a necessary effect of losing time in a strict timing sense.

    When the burst ratio is set to higher than 1.0, faster operations may recover lost time from previously slower operations. This means that any valleys created in the actual op rate of the client can be converted into plateaus of throughput above the strict rate, but only at a speed that fits within (op rate * burst ratio). This allows for workloads to approximate the average target rate over time, with controllable bursting rates. This ability allows for near-strict behavior while allowing clients to still track truer to rate limit expectations, so long as they overall workload is not saturating resources.

    • Method Summary

      Modifier and Type Method Description
      void applyRateSpec​(RateSpec updatingRateSpec)
      Modify the rate of a running rate limiter.
      protected long getNanoClockTime()  
      RateSpec getRateSpec()
      Get the rate spec that this rate limiter was created from.
      long getStartTime()
      Return the system nanoseconds at the time when the last rate change was made active by a starting or restarting rate spec.
      long getTotalWaitTime()
      Return the total number of nanoseconds behind schedule that this rate limiter is, including the full history across all rates.
      long getWaitTime()
      Return the total number of nanoseconds behind schedule that this rate limiter is, but only since the last time the rate spec has changed.
      protected void init​(ActivityDef activityDef)  
      long maybeWaitForOp()
      Block until it is time for the next operation, according to the nanoseconds per op as set by RateLimiter.applyRateSpec(RateSpec)
      long restart()  
      protected void setActivityDef​(ActivityDef def)  
      protected void setLabel​(java.lang.String label)  
      void start()  
      java.lang.String toString()  

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

    • Constructor Detail

      • HybridRateLimiter

        protected HybridRateLimiter()

      • HybridRateLimiter

        public HybridRateLimiter​(ActivityDef def,
                         java.lang.String label,
                         RateSpec rateSpec)

    • Method Detail

      • setLabel

        protected void setLabel​(java.lang.String label)

      • setActivityDef

        protected void setActivityDef​(ActivityDef def)

      • maybeWaitForOp

        public long maybeWaitForOp()
        Description copied from interface: RateLimiter
        Block until it is time for the next operation, according to the nanoseconds per op as set by RateLimiter.applyRateSpec(RateSpec)
        Specified by:
        maybeWaitForOp in interface RateLimiter
        Returns:
        the waittime as nanos behind schedule when this op returns. The returned value is required to be greater than or equal to zero. Note that accuracy of the returned value is limited by timing precision and calling overhead of the real time clock. It will not generally be better than microseconds. Also, some rate limiting algorithms are unable to efficiently track per-op waittime at speed due to bulk allocation mechanisms necessary to support higher rates.

      • getTotalWaitTime

        public long getTotalWaitTime()
        Description copied from interface: RateLimiter
        Return the total number of nanoseconds behind schedule that this rate limiter is, including the full history across all rates. When the rate is changed, this value is check-pointed to an accumulator and also included in any subsequent measurement.
        Specified by:
        getTotalWaitTime in interface RateLimiter
        Returns:
        nanoseconds behind schedule since the rate limiter was started

      • getWaitTime

        public long getWaitTime()
        Description copied from interface: RateLimiter
        Return the total number of nanoseconds behind schedule that this rate limiter is, but only since the last time the rate spec has changed. When the rate is changed, this value is check-pointed to an accumulator and also included in any subsequent measurement.
        Specified by:
        getWaitTime in interface RateLimiter
        Returns:
        nanoseconds behind schedule since the rate limiter was started

      • getRateSpec

        public RateSpec getRateSpec()
        Description copied from interface: RateLimiter
        Get the rate spec that this rate limiter was created from.
        Specified by:
        getRateSpec in interface RateLimiter
        Returns:
        a RateSpec that describes this rate limiter

      • applyRateSpec

        public void applyRateSpec​(RateSpec updatingRateSpec)
        Description copied from interface: RateLimiter
        Modify the rate of a running rate limiter.
        Specified by:
        applyRateSpec in interface RateLimiter
        Parameters:
        updatingRateSpec - The rate and burstRatio specification

      • init

        protected void init​(ActivityDef activityDef)

      • start

        public void start()
        Specified by:
        start in interface Startable

      • restart

        public long restart()

      • getStartTime

        public long getStartTime()
        Description copied from interface: RateLimiter
        Return the system nanoseconds at the time when the last rate change was made active by a starting or restarting rate spec.
        Specified by:
        getStartTime in interface RateLimiter
        Returns:
        long nanoseconds

      • getNanoClockTime

        protected long getNanoClockTime()

      • toString

        public java.lang.String toString()
        Overrides:
        toString in class java.lang.Object