This is the fifth year of Deflect operations and an opportune time to draw some conclusions from the past and provide a round of feedback to our many users and peers. We fought and won several hundred battles with various distributed denial of service and social engineering attacks against us and our clients, expanding the Deflect offerings of open source mitigation solutions to also include website hosting and attack analytics. However, several important missteps were taken to arrive here and this post will concentrate on lessons learned and the way forward in our battle to reduce to prevalence of DDOS as an all too common technique to silence online voices.
Our reflections and this post were motivated by an external evaluation report of the Distributed Deflect service, which you can read in this PDF. The project itself was a technical long shot and an ambitious community building exercise. Lessons learned from this endeavor are summarized within. Its about a 10 minute read :)
During peak times on Deflect throughout 2012-2016 we were serving an average of 3 million unique daily readers and battling with simultaneous DDoS attacks against several clients. The network served websites continuously for the entire 3 1/4 years of project duration, recording less than 30 minutes of down time in total. The project had direct impact on over four hundred independent media, human rights and democracy building organizations.
What we did
eQualit.ie released 10 open source libraries, toolkits and frameworks including tools for network management and DDoS mitigation; a WordPress managed hosting framework; classification and analysis of malicious network behaviour; the Bundler library for website encryption and delivery across an untrusted network, which was also reused in the Censorship.NO project for circumventing Internet filtering infrastructure.
Over three hundred and fifty websites passed through the Deflect protection service. These websites ranged in size and popularity, receiving anything between a dozen daily readers to over a million. Our open door policy meant that websites who had changed their mind about Deflect protection were free to leave and unhindered in any way from doing so. Over the course of the project, we have mitigated over four hundred DDoS attacks and served approximately 1% of Internet users each calendar year (according to our records correlated against Internet World Statistics). Our work also appeared in topical and mainstream media.
Aside from the DDoS protection service, we trained numerous website administrators in web security principles, worked with several small and medium ISPs to set up their own Deflect infrastructure and enabled Internet presence for key organizations and movements involved in national and international events, including the ’13 election in Iran, ’14 elections in Ukraine, Iguala mass kidnapping, Panama papers, and Black Lives Matter among others.
Distributed Deflect
As attacks grew in size, we debated the long-term existence of the project, deciding to prototype an in-kind DDoS mitigation service, whereby websites receiving free protection and any volunteers could join and expand the mitigation network’s size and scope. We wanted to create a service run by the people it protected. The hypothesis envisioned the world’s first participatory botnet infrastructure, whereby the network would be sustained with around a hundred servers run by the Deflect project and several thousand volunteer nodes. Our past experience showed that the best way to mitigate a botnet attack was with a distributed solution, utilizing the design of the Internet to nullify an attack that any single end point/s could not handle by itself. Distributed Deflect brought together people of various background and competencies, blending software development and technical service provision, customer support and outreach, documentation and communications. We designed, prototyped and brought into production core components of a distributed volunteer infrastructure, only to realize that the hypothesis behind our proposal could not scale if we were to maintain the privacy and security of all participants in our network.
An infrastructure that would accept voluntary (untrusted) network resources had to introduce checks for content accuracy and confidentiality, otherwise a malicious node could not only see who was doing what on the Deflect network but delete or change content as it passed through their machine. Our solution was to encrypt web pages as they left the origin server and deliver them to readers as an encrypted bundle, with an additional authentication snippet being sent by another node for verification. Volunteer nodes would only be caching encrypted information and would not be able to replace it with alternative content.
All necessary infrastructure design and software tools to implement this model were built to specification. However, once ready for production and undergoing testing, we realized the error in hypothesis made at the onset. Encrypted bundles grew in size, as all page fonts and various third-party libraries – that make up the majority of web pages today and are usually stored in the browser’s cache – had to be included in each bundle.
This increased network latency and could not scale during a DDoS attack. We were worsening the performance of our infrastructure instead of improving it. Another important factor driving our deliberation was the low cost of server infrastructure. By renting our machines with commercial providers, and using their competitive pricing to our advantage, we have managed to maintain infrastructure costs below 5% of our overall monthly expenditure. Monetary support for a worldwide infrastructure of Deflect servers was not significant when compared with the resources required to service the network. By concentrating development efforts on encrypting and delivering website content from our distributed cache and performance load balancing on a voluntary node infrastructure, we held back work on improving network management and task automation. This meant that the level of entry to providing technical support for the network was set quite high and excluded the participation of technically minded volunteers protected by Deflect.
After several months of further testing, deliberation and consultation with our funders, we decided to abandon the initiative to include voluntary network resources, in favour of continuing the existing mitigation platform and improving its services for clients. As attack mitigation became routine and Deflect successfully defended its clients from relentless DDoS offensives, the team began to look at the impunity currently enjoyed by those launching the attacks. Beginning with a case of a Vietnamese independent media website targeted by bots originating from a state-regulated and controlled Vietnamese ISP, we understood that a story could be extracted from the forensic trail of an attack, that may contain evidence of motivation, method and provenance. If this story could be told, it would give huge advocacy power to the target and begin to peel away at the anonymity enjoyed by its organizers. The cost for attacking Deflectees would raise as exposure and media attention around the event upended the attackers’ goals.
We began to develop an infrastructure that would capture a statistically relevant segment of an attack. Data analysis was achieved through machine-led technology for profiling and classifying malicious actors on our network, visualization tools for human-led investigation and cooperation with peer organizations for tracing activity in our respective networks. This effort became Deflect Labs and in its first twelve months we published three detailed reports covering a series of incidents targeting websites protected by Deflect, exposing their methodology and profiling their networks. Doing some open source intelligence and in collaboration with website staff, we identified a story in each attack exposing possible motivations and identity of the attackers. Following publication and media attention created by these reports, attacks against one of the websites reduced significantly and ceased altogether for the other one.
Bot behavior follows a certain pattern inside the seven dimensional space create by Bothound analytics
Challenges
Many difficulties and problems could be expected with running a high-impact, 24/7 security service for several million daily readers. Fatigue, lack of time for developing new features, round-the-clock emergency coverage and numerous instances of high-stress situations led to burnout and staff turnover. The resources invested in the Distributed Deflect model set back development considerably for other project ambitions.
At around the same time as Deflect was gaining popularity, free mitigation offerings from Cloudflare and Google were introduced in tandem with outreach campaigns targeting independent media and human rights organizations. This led to more options for civil society organizations seeking website protection but made it harder for us to attract the expected number of websites. We started a campaign to define differences in our distinctive approaches to client eligibility, respect for their privacy and clear terms of service, trying a variety of communications and outreach strategies. We were disappointed nonetheless to not have received more support from within our community of peers, as open source solutions and data ownership did not figure highly as criteria for NGOs and media when selecting mitigation options.
… we carry on
Deflect continues to operate and innovate, gradually growing and solidifying. Our ongoing ambitions include offering our clients broader hosting options and coming up with standards and systems for responsible data sharing among like-minded ISPs and mitigation providers. Look out for pleasant graphic user interfaces in our control panels and documentation platforms. We are also prototyping several different approaches to generating revenue in order to sustain the project for the foreseeable future. The goal is to get better without losing track of what we came here to do in the first place. As always, we are here to support our clients’ mission and their right to free expression. We are heartened by their feedback and testimonials.