FlashPanel, a leading enterprise-grade security and management application for Google Apps, makes use of many Google Apps APIs. Each API comes with its own set of rate-limiting quotas, network traffic behavior and response conditions. The challenge we face is to provide our customers with a stable, consistent experience despite these API behavior differences. Fortunately, Google has provided two simple, yet powerful tools to greatly increase our success rate and reliability.
Controlling Throttle Rates with Guava’s RateLimiter
Google Apps APIs operate with quota rates per second or per day. As such, the FlashPanel development team’s objective is to minimize total operation time of a parallelized set of API-intensive processes while avoiding running into Google’s API quota limits. To accomplish this, we can very easily set up a thread-safe RateLimiter for a specific rate per second for each rate restricted API:
final RateLimiter driveApiRateLimiter = RateLimiter.create(QUOTA_RATE_PER_SECOND);
The Guava RateLimiter allows us to attempt to acquire a permit based on the configured rate per second, block until available, and then take it when available, allowing execution.
To use it within the context of a method, we simply reference the rate limiter before making the API call:
public Result performDriveApiCall(driveApiRateLimiter, otherParams){
driveApiRateLimiter.acquire(); // blocks according to rate
// make API call...}This provides us with a very easy solution for rate limiting parallelized calls to the same APIs.
Reducing Impact of Network Traffic Issues Through Configured Backoffs
Occasionally, we experience additional network traffic issues despite enforcing rate limits. Google suggests that API clients use an exponential backoff strategy to handle this variety of error response. Typically this category of network issue resolves itself on its own after fractions of a second, but occasionally more time is needed. All we need to do to receive a successful API response is to simply retry the call some number of times, with the interval of time between retries growing exponentially.
Again, Google has made this easy through the Google HTTP Client for Java library’s ExponentialBackOff builder. This class allows us to configure the initial retry time interval, the exponential multiplier, the maximum total time to attempt to retry, and the maximum total time between retries. For example, we could configure a retry to span five minutes, growing with a factor of two, starting at a one second interval, and growing up to a maximum of one minute between retries. An API call with this configuration would retry with the following pattern in terms of seconds between retries:
1, 2, 4, 8, 16, 32, 60, 60, 60
If after this last retry, the API call still was not successful, the failed http response is returned. The code to configure such a strategy using the ExponentialBackOff.Builder reads as follows:
ExponentialBackOff backoff = new ExponentialBackOff.Builder()
.setInitialIntervalMillis(ONE_SECOND)
.setMultiplier(2.0)
.setMaxIntervalMillis(ONE_MINUTE)
.setMaxElapsedTimeMillis(FIVE_MINUTES)
.build();
One potential “gotcha” that we’ve seen is if we accidentally slam a particular API with many simultaneous API calls. In this event, not only would each of these API calls fail, but they would also schedule their retry strategy to occur simultaneously. All subsequent retries would end up firing simultaneously, causing the API calls to continue to fail due to excess per second volumes. The ExponentialBackoff class accounts for this by including a randomization factor within our retry logic that allows us to have each simultaneous API call stagger at different intervals.
For example, using our previous backoff but now with randomization, one API call may retry with these intervals:
1.04, 1.9, 4.23, 7.8, etc.
While a second API call would retry with something like these intervals:
.98, 2.04, 4.1, 8.15, etc.
With this randomization, we avoid API call sequencing collision, mitigating our chances of encountering quota related errors. To simply add this type of randomization, we append to our builder:
builder.setRandomizationFactor(RANDOMIZATION_FACTOR);
Once we have our exponential backoff strategy configured, the Google HTTP Client for Java library allows us to instantiate and assign an HttpBackOffUnsuccessfulResponseHandler to an HttpRequest as part of the request’s initialization:
private HttpBackOffUnsuccessfulResponseHandler handler = new HttpBackOffUnsuccessfulResponseHandler(backoff);
public void initialize(HttpRequest request){
request.setUnsuccessfulResponseHandler(handler);
}
Final Thoughts
By restricting our API calls using Google Guava’s easy-to-use RateLimiter, and by employing an exponential backoff strategy using the Google HTTP Client for Java library’s ExponentialBackOff, we are able to significantly reduce the amount of network traffic errors received by our Google Apps API calls, improving FlashPanel’s reliability and the overall user experience.
Steve is Senior Architect at BetterCloud, the makers of FlashPanel, the number one security and management application for Google Apps. Follow BetterCloud on Google+ at plus.google.com/+Bettercloud.
Posted by Greg Knoke, Googler
