#Memory leak software#

This is our bug and not the fault of Ragel. These NGINX filter modules parse buffers (blocks of memory) containing HTML responses, make modifications as necessary, and pass the buffers onto the next filter.įor the avoidance of doubt: the bug is not in Ragel itself.

We first used this new parser for the Automatic HTTP Rewrites feature and have been slowly migrating functionality that uses the old Ragel parser to cf-html.īoth cf-html and the old Ragel parser are implemented as NGINX modules compiled into our NGINX builds.

This streaming parser works correctly with HTML5 and is much, much faster and easier to maintain. rl file contains an HTML parser used for all the on-the-fly HTML modifications that Cloudflare performs.Ībout a year ago we decided that the Ragel-based parser had become too complex to maintain and we started to write a new parser, named cf-html, to replace it. Since the very early days of Cloudflare, we’ve used a parser written using Ragel. To modify the page, we need to read and parse the HTML to find elements that need changing. For example, we can insert the Google Analytics tag, safely rewrite links to exclude parts of a page from bad bots, obfuscate email addresses, enable AMP, and more by modifying the HTML of a page. Many of Cloudflare’s services rely on parsing and modifying HTML pages as they pass through our edge servers. This blog post is rather long but, as is our tradition, we prefer to be open and technically detailed about problems that occur with our service. We are grateful that it was found by one of the world’s top security research teams and reported to us.

The greatest period of impact was from February 13 and February 18 with around 1 in every 3,300,000 HTTP requests through Cloudflare potentially resulting in memory leakage (that’s about 0.00003% of requests). We have also not discovered any evidence of malicious exploits of the bug or other reports of its existence. The bug was serious because the leaked memory could contain private information and because it had been cached by search engines. The industry standard time allowed to deploy a fix for a bug like this is usually three months we were completely finished globally in under 7 hours with an initial mitigation in 47 minutes. One of the advantages of being a service is that bugs can go from reported to fixed in minutes to hours instead of months. The team has worked continuously to ensure that this bug and its consequences are fully dealt with. Having a global team meant that, at 12 hour intervals, work was handed over between offices enabling staff to work on the problem 24 hours a day.

#Memory leak software#

At that point it was no longer possible for memory to be returned in an HTTP response.īecause of the seriousness of such a bug, a cross-functional team from software engineering, infosec and operations formed in San Francisco and London to fully understand the underlying cause, to understand the effect of the memory leakage, and to work with Google and other search engines to remove any cached HTTP responses. We quickly identified the problem and turned off three minor Cloudflare features ( email obfuscation, Server-side Excludes and Automatic HTTPS Rewrites) that were all using the same HTML parser chain that was causing the leakage. Cloudflare has always terminated SSL connections through an isolated instance of NGINX that was not affected by this bug. And some of that data had been cached by search engines.įor the avoidance of doubt, Cloudflare customer SSL private keys were not leaked. It turned out that in some unusual circumstances, which I’ll detail below, our edge servers were running past the end of a buffer and returning memory that contained private information such as HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data. He was seeing corrupted web pages being returned by some HTTP requests run through Cloudflare. Last Friday, Tavis Ormandy from Google’s Project Zero contacted Cloudflare to report a security problem with our edge servers.