On March 3rd, starting around 3:00 PM UTC, Sheet Monkey came under a Denial of Service or DoS attack. During this attack our API gateway began rate-limiting our requests at 10,000 requests / second and responding with an error for requests over that limit. The impact was that some users submitting forms to the Sheet Monkey API received an intermittent error instead of the data being correctly added to the sheet. This condition lasted from about 3:00 PM UTC to 7:54 PM UTC, about 4.5 hours.
Timeline
All times are in UTC
- 3:00 PM: DoS attack begins.
- 5:50 PM: We become aware of the service disruption and begin investigation.
- 7:45 PM: We update our firewall configuration to block against the DoS attack.
- 7:54 PM: The firewall configuration completes deployment and the system returns to normal operations.
Impact
Forms would intermittently receive a rate limit error response when submitted. The data would not be added to the Google Sheet when this error occurred.
Next Steps
- Implement an endpoint monitoring solution so we can respond more quickly to api disruptions.
- Update our firewall configuration to intelligently detect denial of service attacks rather than using rate limiting.
To our customers, we are deeply sorry that this incident occurred and that it prevented you from saving data in your sheets. We have always prided ourselves on the stability and reliability of our little service and today we did not live up to our standards. Actions speak louder than words and we will continue to improve Sheet Monkey and work to make our service more resilient against attacks.