BotoServerError - 400 Bad Request - Rate Exceeded

This is for anyone out there using python-boto with Amazon CloudSearch, who is getting a throttle limit and googling around for an answer. You’re probably getting an error like this:

<ErrorResponse xmlns=“http://cloudsearch.amazonaws.com/doc/2011-02-01/”>
  <Error>
    <Type>Sender</Type>
    <Code>Throttling</Code>
    <Message>Rate exceeded</Message>
  </Error>
  <RequestId>c753f49e-26b9-11e3-a879-0d3b8f7ade79</RequestId>
</ErrorResponse>

Here’s what’s happening: Although there aren’t any limits on the volume of searches you can make with CloudSearch, there are throttling limits on the preliminary calls to make to get the document & search service.

This can be solved by caching the search service object. Here’s an example with Django’s caching:

import boto
django_cache

cs_conn = boto.connect_cloudsearch(
    aws_access_key_id=YOUR_KEY_ID,
    aws_secret_access_key=YOUR_ACCESS_KEY,
)

search_kwargs = {}
search_kwargs['q'] = "Stuff to find"

try:
    search_service = django_cache.get("cloudsearch_search_service")
    # Need to try to execute something for it to fail         
    results = search_service.search(**search_kwargs)
except:
    domain = cs_conn.lookup("domain-name")
    document_service = domain.get_document_service()
    search_service = document_service.get_search_service()
    django_cache.set("cloudsearch_search_service", search_service, 300)
    results = search_service.search(**search_kwargs)

print results.doc

Written on November 22, 2013