Distributed Deflect – project review

This is the fifth year of Deflect operations and an opportune time to draw some conclusions from the past and provide a round of feedback to our many users and peers. We fought and won several hundred battles with various distributed denial of service and social engineering attacks against us and our clients, expanding the Deflect offerings of open source mitigation solutions to also include website hosting and attack analytics. However, several important missteps were taken to arrive here and this post will concentrate on lessons learned and the way forward in our battle to reduce to prevalence of DDOS as an all too common technique to silence online voices.

Our reflections and this post were motivated by an external evaluation report of the Distributed Deflect service, which you can read in this PDF. The project itself was a technical long shot and an ambitious community building exercise. Lessons learned from this endeavor are summarized within. Its about a 10 minute read :)

During peak times on Deflect throughout 2012-2016 we were serving an average of 3 million unique daily readers and battling with simultaneous DDoS attacks against several clients. The network served websites continuously for the entire 3 1/4 years of project duration, recording less than 30 minutes of down time in total. The project had direct impact on over four hundred independent media, human rights and democracy building organizations.

final_report_graphWhat we did

eQualit.ie released 10 open source libraries, toolkits and frameworks including tools for network management and DDoS mitigation; a WordPress managed hosting framework; classification and analysis of malicious network behaviour; the Bundler library for website encryption and delivery across an untrusted network, which was also reused in the Censorship.NO project for circumventing Internet filtering infrastructure.

Over three hundred and fifty websites passed through the Deflect protection service. These websites ranged in size and popularity, receiving anything between a dozen daily readers to over a million. Our open door policy meant that websites who had changed their mind about Deflect protection were free to leave and unhindered in any way from doing so. Over the course of the project, we have mitigated over four hundred DDoS attacks and served approximately 1% of Internet users each calendar year (according to our records correlated against Internet World Statistics). Our work also appeared in topical and mainstream media.

Aside from the DDoS protection service, we trained numerous website administrators in web security principles, worked with several small and medium ISPs to set up their own Deflect infrastructure and enabled Internet presence for key organizations and movements involved in national and international events, including the ’13 election in Iran, ’14 elections in Ukraine, Iguala mass kidnapping, Panama papers, and Black Lives Matter among others.

Distributed Deflect

As attacks grew in size, we debated the long-term existence of the project, deciding to prototype an in-kind DDoS mitigation service, whereby websites receiving free protection and any volunteers could join and expand the mitigation network’s size and scope. We wanted to create a service run by the people it protected. The hypothesis envisioned the world’s first participatory botnet infrastructure, whereby the network would be sustained with around a hundred servers run by the Deflect project and several thousand volunteer nodes. Our past experience showed that the best way to mitigate a botnet attack was with a distributed solution, utilizing the design of the Internet to nullify an attack that any single end point/s could not handle by itself. Distributed Deflect brought together people of various background and competencies, blending software development and technical service provision, customer support and outreach, documentation and communications. We designed, prototyped and brought into production core components of a distributed volunteer infrastructure, only to realize that the hypothesis behind our proposal could not scale if we were to maintain the privacy and security of all participants in our network.

ddeflect

An infrastructure that would accept voluntary (untrusted) network resources had to introduce checks for content accuracy and confidentiality, otherwise a malicious node could not only see who was doing what on the Deflect network but delete or change content as it passed through their machine. Our solution was to encrypt web pages as they left the origin server and deliver them to readers as an encrypted bundle, with an additional authentication snippet being sent by another node for verification. Volunteer nodes would only be caching encrypted information and would not be able to replace it with alternative content.

bundler

All necessary infrastructure design and software tools to implement this model were built to specification. However, once ready for production and undergoing testing, we realized the error in hypothesis made at the onset. Encrypted bundles grew in size, as all page fonts and various third-party libraries – that make up the majority of web pages today and are usually stored in the browser’s cache – had to be included in each bundle.

This increased network latency and could not scale during a DDoS attack. We were worsening the performance of our infrastructure instead of improving it. Another important factor driving our deliberation was the low cost of server infrastructure. By renting our machines with commercial providers, and using their competitive pricing to our advantage, we have managed to maintain infrastructure costs below 5% of our overall monthly expenditure. Monetary support for a worldwide infrastructure of Deflect servers was not significant when compared with the resources required to service the network. By concentrating development efforts on encrypting and delivering website content from our distributed cache and performance load balancing on a voluntary node infrastructure, we held back work on improving network management and task automation. This meant that the level of entry to providing technical support for the network was set quite high and excluded the participation of technically minded volunteers protected by Deflect.

ddeflect-headers-response

After several months of further testing, deliberation and consultation with our funders, we decided to abandon the initiative to include voluntary network resources, in favour of continuing the existing mitigation platform and improving its services for clients. As attack mitigation became routine and Deflect successfully defended its clients from relentless DDoS offensives, the team began to look at the impunity currently enjoyed by those launching the attacks. Beginning with a case of a Vietnamese independent media website targeted by bots originating from a state-regulated and controlled Vietnamese ISP, we understood that a story could be extracted from the forensic trail of an attack, that may contain evidence of motivation, method and provenance. If this story could be told, it would give huge advocacy power to the target and begin to peel away at the anonymity enjoyed by its organizers. The cost for attacking Deflectees would raise as exposure and media attention around the event upended the attackers’ goals.

We began to develop an infrastructure that would capture a statistically relevant segment of an attack. Data analysis was achieved through machine-led technology for profiling and classifying malicious actors on our network, visualization tools for human-led investigation and cooperation with peer organizations for tracing activity in our respective networks. This effort became Deflect Labs and in its first twelve months we published three detailed reports covering a series of incidents targeting websites protected by Deflect, exposing their methodology and profiling their networks. Doing some open source intelligence and in collaboration with website staff, we identified a story in each attack exposing possible motivations and identity of the attackers. Following publication and media attention created by these reports, attacks against one of the websites reduced significantly and ceased altogether for the other one.

Bot behavior follows a certain pattern inside the seven dimensional space create by Bothound analytics

Bot behavior follows a certain pattern inside the seven dimensional space create by Bothound analytics

Challenges

Many difficulties and problems could be expected with running a high-impact, 24/7 security service for several million daily readers. Fatigue, lack of time for developing new features, round-the-clock emergency coverage and numerous instances of high-stress situations led to burnout and staff turnover. The resources invested in the Distributed Deflect model set back development considerably for other project ambitions.
At around the same time as Deflect was gaining popularity, free mitigation offerings from Cloudflare and Google were introduced in tandem with outreach campaigns targeting independent media and human rights organizations. This led to more options for civil society organizations seeking website protection but made it harder for us to attract the expected number of websites. We started a campaign to define differences in our distinctive approaches to client eligibility, respect for their privacy and clear terms of service, trying a variety of communications and outreach strategies. We were disappointed nonetheless to not have received more support from within our community of peers, as open source solutions and data ownership did not figure highly as criteria for NGOs and media when selecting mitigation options.

… we carry on

Deflect continues to operate and innovate, gradually growing and solidifying. Our ongoing ambitions include offering our clients broader hosting options and coming up with standards and systems for responsible data sharing among like-minded ISPs and mitigation providers. Look out for pleasant graphic user interfaces in our control panels and documentation platforms. We are also prototyping several different approaches to generating revenue in order to sustain the project for the foreseeable future. The goal is to get better without losing track of what we came here to do in the first place. As always, we are here to support our clients’ mission and their right to free expression. We are heartened by their feedback and testimonials.

Read More

Creating a Hosts File Entry

If you wish to access your domain before your DNS has been updated, you can update your local ‘hosts file’, which will allow your computer to view your new site. Follow the appropriate instructions below.

Please note that this will work only with HTTPS and not with HTTP.

If you need any help with this procedure (for example because nslookup is not installed in your system and you can’t figure out what the IP of your SFTP server is), we are ready to help: please contact us through the Dashboard or send us an email.

OS X:

  1. Open Terminal
  2. Launch the following command (replacing SFTP_host with the address of your SFTP host you received in your activation email):

    $ nslookup SFTP_host

  3. The result will be something like the following output. The last line contains the IP address of your SFTP host, which you will need to add to your hosts file (numbers arranged in this form: XX.XX.XX.XX).

    Server:        YY.ZZ.XX.ZZ
    Address:    YY.ZZ.XX.ZZ#53

    Non-authoritative answer:
    Name:    grwtrcweg.deflect.ca
    Address: XX.XX.XX.XX

  4. Type ‘sudo nano /private/etc/hosts’
  5. Press Ctrl+Shift+V to take you to the end of the file
  6. Enter the text ‘XX.XX.XX.XX <yourdomain>’ (replacing `XX.XX.XX.XX` with the actual IP of your SFTP host and <yourdomain> with the URL of your website).
  7. Press Ctrl+x to exit
  8. Press y to save

Alternatively you can download the Hosts preference pane helper from here: https://github.com/specialunderwear/Hosts.prefpane/downloads

Windows:

  1. Launch the Command Prompt and enter:

    C:\>nslookup example.com

    whereby you need to replace example.com with your SFTP host address.

  2. The result will contain the IP address of your SFTP host, which you will need to add to your hosts file (numbers arranged in this form: XX.XX.XX.XX).

    Address: XX.XX.XX.XX

  3. Click “Start” button
  4. Click “All Programs”
  5. Click “Accessories”
  6. Right-click on Notepad and then click Run as administrator.
  7. If you are prompted for an administrator password or for a confirmation, type your password, or click Allow/Yes.
  8. Open the Hosts file. Discover the location for your version of windows here (https://en.wikipedia.org/wiki/Hosts_(file)#Location_in_the_file_system)
  9. Enter the text ‘XX.XX.XX.XX <yourdomain>’ (replacing `XX.XX.XX.XX` with the actual IP of your SFTP host and <yourdomain> with the URL of your website).
  10. Click Save on the Edit menu. (If using Windows 7, you will need to click Save on the File menu.)

Linux:

  1. Open a terminal.
  2. Launch the following command (replacing SFTP_host with the address of your SFTP host you received in your activation email):

    $ nslookup SFTP_host

  3. The result will be something like the following output. The last line contains the IP address of your SFTP host, which you will need to add to your hosts file (numbers arranged in this form: XX.XX.XX.XX).

    Server:        YY.ZZ.XX.ZZ
    Address:    YY.ZZ.XX.ZZ#53

    Non-authoritative answer:
    Name:    grwtrcweg.deflect.ca
    Address: XX.XX.XX.XX

  4. Open the file /etc/hosts with vim or your favourite editor as root:

    $ sudo vim /etc/hosts

  5. Add the following line, replacing `XX.XX.XX.XX` with the IP address of your SFTP host, `example.com` with the URL of your website and `example` with the name of your website:

    XX.XX.XX.XX example.com example

  6. Ensure that the nsswitch.conf file is correct. The nsswitch.conf file controls in which order services will be consulted for name service lookups, in our case we are looking for the “hosts” service:

    $ grep host /etc/nsswitch.conf hosts: files dns

    Check that “files” comes before “dns”. If it doesn’t, edit the file to obtain the above result.

  7. Check that your changes produced the wanted effect with this command:

    $ ping -c 1 example.com

    The result should be something like this (with XX.XX.XX.XX being replaced by the IP of your SFTP host):

    PING example.com (XX.XX.XX.XX) 56(84) bytes of data.

Read More

How to Flush Your Local DNS Resolver’s Cache

If your computer cannot reach a certain website this could be because your local DNS resolver’s cache contains an outdated record. For example, you updated your DNS records to point to eQPress but instead you are seeing your old website. This is when flushing your DNS cache will speed things up.

Mac (OS X)

In the Command Terminal, type one of these commands:

sudo killall -HUP mDNSResponder
sudo discoveryutil udnsflushcaches

sudo dscacheutil -flushcache
sudo lookupd -flushcache

Windows

Run the following command in a Command Prompt window:

ipconfig /flushdns

Read More

Recommendations for Improving Your WordPress SEO

When it comes to search engine optimization (SEO), choosing the right WordPress theme framework becomes critical. Genesis does a great job of doing all the right things for search engine optimization (SEO). You will want to add some kind of analytics tracking to your site so you can gain insight about who is visiting your site and how your site is being used. Typically most people use Google Analytics, but if you are considering to use it in your website, please consider that eQPress Console already shows you some statistics and that Google Analytics, as well as other tools for website statistics, track users and can violate your readers’ privacy. If you decide to use one of these tools, Genesis provides a field on the Theme Settings page (wp_footer) to enter your analytics tracking code.

There are other great theme frameworks that are just as effective as Genesis, but if you choose not to use a framework or prefer to build your own theme, then you should use a plugin such as wordpress-seo by Yoast and use that to further optimize your pages and posts for SEO. The plugin has a ton of options which can be a bit overwhelming, but typically the defaults are fine. The plugin will also analyse your pages and give you recommendations on how to improve your content, title and other aspects of the page to make it better for SEO. There are lots of tutorials on using the plugin and of course the author of the plugin is a great source for learning about SEO: https://yoast.com

A couple more recommendations: in addition to a Google Analytics account for your site, you should also create a Google Webmaster Tools account and link it to your analytics account. And the other thing is to create a sitemap.xml file. The search engine crawlers look for that to more accurately index your site. The wordpress-seo plugin will create one for you, but there are simpler plugins to get this task done such as google-sitemap-plugin.

Read More

Choosing a Canonical Website Address

Canoni-what?

Canonical is the word used to describe the one address that you want the world to go to when they look you up. The typical choices are whether to use www in front of your domain or not. The classic example follows:

http://www.example.com/

or

http://example.com/

Choosing what your canonical website address (URL) will be is totally up to you. It’s a preference and there’s no right answer. As you can see by looking up at your browser’s location bar now, eQualit.ie has chosen a URL without www. If you start taking notice of the other websites you visit, you’ll probably see that there’s no regular pattern. Google chooses www. The wordpress.org team chooses non-www. It really doesn’t matter. What does matter is making that choice early and sticking with it.

Considering the Apex

One unique factor (with respect to hosting on) in your decision-making process is whether or not your domain will be hosted by a DNS company that supports pointing the non-www (officially called the apex record) address to a CNAME. If your DNS host does not support this feature, we recommend you choose www to be your canonical website address.

References

Here’s an article at Google Webmaster Tools called Use Canonical URLs that will help you to learn more about their view of canonical URLs.

Also, Matt Cutts provides some very helpful insight and a FAQ about SEO and URL canonicalization.

Read More

First Steps with Your eQPress Site

Your shiny new eQPress site is ready to go! Now what? Here are some recommendations.

  1. Enable “pretty” permalinks under “Settings” -> “Permalinks”.
    Typically “Post name” is a good option, but you can choose whichever setting you prefer other than “Plain”. The reason for doing this is two-fold: on the one hand, your URLs just look nicer, on the other hand this could also increase your performance because this type of URL has better chances of getting cached.
  2. Next install a plugin to protect your website against comment spam. Anti-spam is easy to use, needs no configuration and just works.

Now that you’ve made some initial steps, you can take some time to read the official guide: First Steps with WordPress

Read More

Learning WordPress

Here are some great resources for learning how to use WordPress.

The Official WordPress User Manualhttps://make.wordpress.org/support/user-manual/
This is a living document created and maintained by the amazing and dedicated WordPress.org team.

Easy WP Guide – https://easywpguide.com/
You won’t find any talk of HTML, PHP or creating WP Themes here. What you will find is an easy to follow WordPress manual that will help you understand the basics of editing your site content.

Codex for the WordPress Projecthttp://codex.wordpress.org/
The online manual for WordPress and a living repository for WordPress information and documentation.

WordPress TVhttp://wordpress.tv/
Videos from WordCamps and more about our favourite CMS and blogging platform.

WP 101http://www.wp101.com/
Some free but mostly pay access video tutorials about learning the ins and outs of WordPress.

Read More

Restricting Access to Your Website

There are times when you want to prevent the world from seeing your website. One reason might be that you are in the process of designing and developing and don’t want to share your progress with the world, since it might not be ready for all eyes. Another reason might be that you only want registered users to have access to view your content. The following plugins can help restrict access to your WordPress website:

Ultimate Coming Soon Page – http://wordpress.org/plugins/ultimate-coming-soon-page/

Restrict Site Access – http://wordpress.org/plugins/restricted-site-access/

Read More

Debugging the Dreaded “White Screen of Death”

If you are here because you recently installed a plugin, then take a look at this flow chart.

Both PHP errors and database errors can manifest as a white screen, a blank screen with no information, commonly known in the WordPress community as the WordPress White Screen of Death (WSOD).  Here are some basic steps you can follow to begin debugging this problem if it’s not been caused by a new plugin that you’ve just installed:

  1. SFTP to your document root (contact us through the Dashboard or send us an email if you’ve lost your SFTP credentials).
  2. Change into the wordpress directory.
  3. View or download the php-errors.log file.
  4. Take a look at the last few lines of the error log to determine the cause of the white screen.

You might see a very specific PHP error which will provide a line number in the file that’s causing the problem. At this point you can:

  1. Download the file that’s causing the problem.
  2. Go to the line that was listed in the error log.
  3. Fix the issue.
  4. Upload the file back to the server.
  5. Test your website.

If the white screen does not go away, repeat all of the steps above. Sometimes there could be more than one error.

Read More