We Chat (But Not about Everything)

Imagine if your favourite social media application silently censored your posts, but gave you no information about what topics are censored.

Imagine if everything seemed fine as you posted message after message and image after image, for days on end with no issues, but then occasionally one of your posts would simply not appear without explanation.

And what if the messages or images you are prevented from posting sometimes seem connected with a controversial political issue, but other times not?  Perhaps it’s deliberate, you might guess. Perhaps it’s just you and your bad Internet connection?  Who can say for sure?

Unfortunately this Kafka-esque situation is the reality for well over a billion users of WeChat and Sina Weibo, two of China’s largest social media applications and among the largest in the world.

Our new report provides detailed evidence from systematic experiments we have been performing on WeChat and Sina Weibo to uncover censorship on each of the applications.  As with prior reports on each of the applications, we are interested in enumerating censored topics — a difficult question to answer since neither of the companies is transparent about what they block.

For our latest research, we focused on censorship of discussions about the so-called “709 Crackdown.” This crackdown refers to the nationwide targeting by China’s police of nearly 250 human rights lawyers, activists, as well as some of their staff and family, since July 9, 2015, when lawyers Wang Yu (王宇) and her husband Bao Longjun (包龙军) were forcibly “disappeared.”  The 709 Crackdown is considered one of the harshest systematic measures of repression on civil society undertaken by China since 1989, and is the subject of much ongoing international media and human rights discussion.  

Unfortunately, as our experiments show, a good portion of that discussion fails to reach Chinese users of WeChat and Weibo. Our research shows that certain combinations of keywords, when sent together in a text message, are censored. When sent alone, they are not.  So, for example, if one were to text 中国大陆 (Mainland China) or 王全璋的妻子 (Wang Quanzhang’s Wife) or 家属的打压 (Harassment on Relatives) individually, the messages would get through.  Sent together, however, the message would be censored.  The Citizen Lab’s Andrew Hilt’s has created a visualization showing these keyword combinations here: https://citizenlab.org/709crackdownviz

In addition to a large number of censored keyword combinations our tests unearthed, we also discovered 58 images related to the 709 Crackdown that were censored on WeChat Moments for accounts registered with a mainland China phone number. (For accounts registered with a non-mainland China phone number, on the other hand, the images and keyword combinations go through fine). This is the first time we have documented censorship of images on a social media platform, and we are continuing to investigate the exact mechanism by which it takes place.

The purpose of Citizen Lab’s research on applications like WeChat and Weibo is to better understand and bring transparency to restrictions such as these. We live in a world in which our choices and decisions are increasingly determined by algorithms buried in the applications we use.  What websites we visit, with whom we communicate, and what we say and do online are all increasingly determined by these code-based rules.  Whether those algorithms are fair or not, whether they respect human rights, whether they make mistakes or not, are all questions that can only be answered if the algorithms can be properly examined.

Unfortunately, many social media hide their algorithms, either for proprietary and financial reasons (they want to protect the “secret sauce” that earns them money) or for political reasons (their algorithms are used to enforce restrictions on speech and they don’t want their customers to know about it).  Our research aims to break through that obfuscation and bring such algorithms to account.

Generally speaking, the algorithms that drive social media censorship or surveillance can operate in one of two ways: either on the client side — meaning, inside the application on your device; or on the server side — meaning, inside one of the company’s computers that runs the service.  Typically, to investigate the former, we rip the application apart — “reverse engineer” it — and subject it to various tests to determine what the algorithm does beneath the surface.

For server-side rules, on the other hand, whatever censorship or surveillance is going on happens inside the company’s infrastructure, making it more challenging to interrogate the rules.  Both WeChat and Weibo perform censorship and surveillance on the server side, so we had to undertake detailed experiments using combinations of keywords and images drawn from news stories and fed into the applications systematically to zero in on what’s filtered.  You can read about these experiments in the full report here: https://citizenlab.org/2017/04/we-cant-chat-709-crackdown-discussions-blocked-on-weibo-and-wechat/

Our report serves as a reminder that for a large portion of the world, social media act as gatekeepers of what they can read, speak, and see. When they operate in a repressive environment like China, social media can end up surreptitiously preventing important political topics from being discussed.  Our finding that WeChat is now also systematically censoring images as well as text opens up the daunting prospect of multi-media censorship and surveillance on social media.

Taming the “Wild West” Commercial Spyware Market

Today, my colleague Sarah McKune and I co-authored an article, entitled “Who’s Watching Little Brother? A Checklist for Accountability in the Industry Behind Government Hacking.”  A blog post about the report can be found here, and the article is available in PDF here.

The report outlines a “checklist” for regulating the commercial spyware market.  As we have reported on numerous occasions as part of Citizen Lab’s research, there is ample evidence of growing abuses surrounding the commercial spyware market. In spite of the pledges made by some in the industry — that self-regulation works, that they are just following “local laws” — we have shown how companies like Finfisher, Hacking Team, and NSO Group supply their products and services to governments that use them to target journalists, human rights defenders, and even anti-obesity activists. We have tracked the proliferation of some of these services to some of the world’s most autocratic regimes.  It is obvious that these abuses are going to grow unless something is done to mitigate these trends.

Unfortunately, debate until now about what to do about these abuses has revolved in binary form around either export controls or an unregulated wild west.  In our article, we develop instead a checklist for a “web of constraints” around the industry that involves multiple strategies and different mechanisms, including application of existing laws.  We hope that these checklist provides a helpful roadmap for policymakers and others who want to do something about the excesses of this industry and we look forward to feedback.

Read the article here: https://citizenlab.org/wp-content/uploads/2017/03/citizenlab_whos-watching-little-brother.pdf [PDF]



Mexico, NSO Group, and the Soda Tax

I am pleased to announce a new Citizen Lab report, entitled “Bitter Sweet: Supporters of Mexico’s Soda Tax Targeted With NSO Exploit Links,” authored by John Scott-Railton, Bill Marczak, Claudio Guarnieri, and Masashi Crete-Nishihata.

The full report is here:  https://citizenlab.org/2017/02/bittersweet-nso-mexico-spyware/

New York Times has an exclusive here: https://www.nytimes.com/2017/02/11/technology/hack-mexico-soda-tax-advocates.html

In recent years, the research of the Citizen Lab and others has revealed numerous disturbing cases involving the abuse of commercial spyware: sophisticated products and services ostensibly restricted in their sales to government clients and used solely for legitimate law enforcement.

Contrary to what companies like Hacking Team, Gamma Group, NSO Group and others claim about proper industry self regulation, we have repeatedly uncovered examples where governments have used these powerfully invasive tools to target human rights defenders, journalists, and legitimate political opposition.

To this list, we can now add research scientists and health advocates.

The “Bitter Sweet” case has its origins in a prior Citizen Lab investigation — our Million Dollar Dissident report, in which we found that a UAE-based human rights defender, Ahmed Mansoor, was targeted by UAE authorities using the sophisticated “Pegasus” spyware suite, sold by Israeli cyber warfare company, NSO Group.

As part of that report, we published technical indicators — essentially digital signatures associated with the NSO Group’s infrastructure and operations — and encouraged others to use them to find evidence of more targeting.  When we published our report in August 2016, we knew there was at least one Mexican targeted — a journalist — and so suspected there might be some targeting there.

Shortly after the publication of our report, Citizen Lab was contacted by Access Now, which had received a request for assistance on its digital helpline from two Mexican NGOs working on digital rights and security, R3D and SocialTIC.  Together, we worked to track down suspicious messages received by Mexicans, which led us to the Bitter Sweet case.

The title of our report refers to the fact that all of those whom we found targeted in this campaign were involved in a very high-profile “soda tax” campaign in Mexico. A soda tax is part of an anti obesity effort to add taxes to lower consumption of sugary drinks and sodas.  Although many in Mexico are behind the campaign, some in the beverage industry and their stakeholders are obviously not.

In the midst of controversy around the soda tax campaign, at least three prominent research scientists and health advocates received similar (in some cases, identical) suspicious SMS messages that included telltale signs of NSO Group’s attack infrastructure. Had any of them clicked on the links, their iPhones would have been silently compromised, allowing the perpetrators to listen in on their calls, read their emails and messages, turn on their camera, and track their movements — all without their knowledge.

What is most remarkable about the targeting are the steps the perpetrators took to try to trick the scientists and advocates to click on the links.  For example, one of the targets, Dr. Simon Barquera, a well respected researcher at the Mexican Government’s Instituto Nacional de Salud Pública, received a series of increasingly inflammatory messages.  The first SMSs concerned fake legal cases in which the scientist was supposedly involved.  Those following got more personal: a funeral, allegations his wife was having an affair (with links to alleged photos), and then, most shocking, that his daughter, who was named in the SMS, had been in an accident, was in grave condition, and that Dr. Barquera should click a link to see which hospital emergency room into which she was admitted.

While we can’t attribute this campaign to a particular company or government agency, it is obvious those behind the targeting have a stake in getting rid of the soda tax, and that points to the beverage industry and their investors and backers in the Mexican government. It is important to point out that Mexico is on record purchasing NSO Group’s services and NSO Group itself asserts it only sells to legitimate government representatives.  But clearly the NSO’s “lawful intercept” services are not being used in Mexico to fight crime or hunt terrorists, unless those who are advocating against obesity are considered criminal terrorists. We feel strongly that both the Mexican and the Israeli governments (the latter approves exports of NSO products) undertake urgent investigations.

Finally, our report shows the value of careful documentation of suspicious incidents, and ongoing engagement between researchers, civil society organizations, and those who are targeted by malicious actors who wish to do harm.  The epidemic of targeted digital attacks facing civil society will require an all-of-society defence.  The cooperation shown on this investigation by Citizen Lab researchers, Access, R3D, and SocialTIC is a model of how it can be done.

The Easy and Affordable Way to Undertake Cyber Espionage

I am pleased to announce a new Citizen Lab report, entitled “Nile Phish: Large-Scale Phishing Campaign Targeting Egyptian Civil Society,” authored by the Citizen Lab’s John Scott-Railton, Bill Marczak, and Etienne Maynier, in collaboration with Ramy Raoof of the Egyptian Initiative for Personal Rights.

The full report is here:  https://citizenlab.org/2017/02/nilephish-report/

When most of us think of state cyber espionage, what likely comes to mind are extraordinary technological capabilities: rare un-patched software vulnerabilities discovered by teams of highly skilled operators, or services purchased for millions from shadowy “cyber warfare” companies.  To be sure, some cyber espionage fits this description, as any perusal through the Snowden disclosures or our recent “Million Dollar Dissident” report will show. But not all of them do.  More often than not, cyber espionage can be surprisingly low-tech and inexpensive, and yet no less effective, than the glitzy stereotypes.

The Egyptian “Nile Phish” campaign is a case in point.

An authoritarian country racked with domestic insecurity and political turmoil, the Egyptian government has mounted a growing crackdown on civil society.  Part of that crackdown involves investigations of alleged “foreign funding” of Egyptian NGOs — known within Egypt as “Case 173.”

Beginning in November 2016, Egyptian NGOs and their staff under Case 173 investigation simultaneously began receiving identical, legitimate looking emails in their inboxes.  Fortunately, technical staff at one such NGO, the Egyptian Initiative for Personal Rights, suspected something wasn’t right, and reached out to us at the Citizen Lab for further investigation.

With EIPR’s assistance, we began analyzing the suspicious emails and discretely contacting other Egyptian organizations and individuals who received them.  What we discovered was an elaborate, coordinated, and multi-phased “phishing” campaign in which legitimate looking emails are sent to unsuspecting users in an attempt to trick them into entering their passwords into fraudulent websites controlled by the operators.

If this type of activity sounds familiar, it is because phishing is widely used as a tactic in the world of everyday cyber crime.  Just yesterday, I received a warning from the University of Toronto’s IT support unit about a malicious email sent to faculty and staff with a notice about a non-existent “Campus Security Notification.”  It may also sound familiar because it was precisely this type of phishing tactic that Russian hackers used to compromise the gmail account of the chairman of the 2016 Hillary Clinton campaign, John Podesta (illustrating the principle that even Great Powers sometimes pick cheap seats as long as it gets them where they want to go).

In the case of #NilePhish, Egyptian NGOs and individuals received emails with an invitation to attend a workshop about Case 173.  The operators used language from a real NGO statement that had been circulating among the community, and included as co-sponsors some of the very NGOs that were targeted.  A second wave of phishing emails included what purported to be a list of individuals subject to a travel ban under Case 173 (who among Egyptian civil society wouldn’t be tempted to check if they were included on that list?).  Alongside these carefully crafted emails — and seemingly just to mix things up — generic phishing attempts were sent with email security or fake courier delivery notifications.

Led by John Scott Railton, our team analyzed the emails and the server infrastructure in detail.  Dozens of fake but legitimate sounding domains were used by the operators to host websites that appeared to be Dropbox login pages or Gmail “failed login” warning messages.  Emails were sent from addresses like fedex_tracking[@]outlook.sa and dropbox.notfication[@]gmail.com.

Because of mistakes made on the part of the attackers, and our team’s use of multiple data sources and methods that are outlined in the report, we were able to eventually link more than 90 messages sent to seven NGOs and individuals as part of a single concerted campaign.  While we were unable to definitively attribute the campaign to an Egyptian government agency, strong circumstantial evidence exists that support it.  For example, we observed phishing against the colleagues of the Egyptian lawyer Azza Soliman, within hours of her arrest in December 2016. The phishing claimed to be a copy of her arrest warrant.  It is highly unlikely a random cyber criminal would be privy to such details, but quite likely someone connected to her arrest is.

Phishing may be an example of “poor man’s” cyber espionage, but the reason it’s used by everyone from Ukrainian securities fraudsters to Russian hackers to para-state groups is because it works.   From a government perspective, why bother with expensive wire transfers, complicated end user license agreements, third party resellers, and export controls, when a handful of cleverly constructed emails and websites will do the job?

The flip side is that there are cheap and easy ways to defend against phishing: users can be educated not to click on links or open emails that look legitimate and to spot giveaways of their malicious nature; tech companies can put in place two-factor authentication for access to their services by default; and NGOs can employ dedicated technologists who can manage their networks and alert their staff to the latest alerts.

Fortunately for Egyptian civil society, EIPR is just such an organization.

#NilePhish is ongoing, and we strongly suspect that there may be other targets of this campaign we have not yet identified.  We hope that the detailed indicators we are publishing can be used by systems administrators and others to find more evidence of targeting and alert potential victims.

Read the full report here: https://citizenlab.org/2017/02/nilephish-report/

Read EIPR’s report on #NilePhish in Arabic: http://eipr.org/nilephish

The DHS/FBI Report on Russian Hacking was a Predictable Failure

Russian cyber espionage against American political targets has dominated the news in recent months, intensifying last week with President Barack Obama’s announcement of sanctions against Russia.

Cyber espionage is, of course, nothing new. But using data collected in cyber espionage operations to interfere in the U.S. election process on behalf of one of the candidates — one who appears to be smitten with Russian President Vladimir Putin — is a brazen and unprecedented move that deserves a firm political response from the U.S. government on behalf of the public interest.

The expulsion of 35 Russian diplomats, the shutting down of two Russian-owned estates the US claims were used for intelligence activities, and the targeted financial sanctions on Russian individuals and organizations all show the Obama administration understands at least part of what such a firm response should entail.

Unfortunately, the White House was unable to produce the most critical part for the credibility of their action: that to be politically effective in today’s Internet age, such a response also needs to be backed up with solid evidence. Here, the administration failed miserably, but also predictably. And it’s not necessarily because it doesn’t have the evidence. Instead, the U.S. government simply failed to present it.

My latest piece is an analysis of the DHS/FBI report on Russian cyber espionage, published in Just Security.  Read the entire piece here:

WeChat: “One App, Two Systems”

Days are long gone when we used to interact with the Internet as an undifferentiated network. The reality today is that what we communicate online is mediated by companies that own and operate the Internet services we use.  Social media in particular have become, for an increasing number of people, their windows on reality.  Whether, and in what ways, those windows might be distorted — by corporate practices or government directives — is thus a matter of significant public importance (but not always easy to discern with the naked eye).

Take the case of WeChat — the most popular chat application in China, and the fourth largest in the world with 806 million monthly active users.  WeChat is more than just an instant messaging application. It is more like a lifestyle platform.  WeChat subscribers use the app not only to send text, voice, and video but to play games, make mobile payments, hail taxis, and more.

As with all other Internet services operating in China, however, WeChat must comply with extensive government regulations that require companies to police their networks and users, and share user data with security agencies upon request.  Over numerous recent case-study reports, Citizen Lab research has found that many China-based applications follow these regulations by building into their applications hidden keyword censorship and surveillance.  WeChat is no exception, although with a twist.

Today, we are releasing a new report, entitled “One App, Two Systems: How WeChat uses one censorship policy in China and another internationally.  For this report, we undertook several controlled experiments using combinations of China, Canada, and U.S. registered phone numbers and accounts to test for Internet censorship on WeChat’s platform.  What we found was quite surprising.

Turns out that there is substantial censorship on WeChat, but split along several dimensions.  There is keyword filtering for users registered with a mainland China phone number but not for those registering with an international number.  However, we also found that once a China-based user had registered with a mainland China phone number, the censorship follows them around — even if they switch to an international phone number, or work, travel, or study abroad.  To give some context, there are roughly 50 million overseas Chinese people working and living abroad.  China’s “One-App, Two Systems” keeps them under the control of China’s censorship regime no matter where they go. This extra-territorial application of information controls is quite unique, and certainly a disturbing precedent to set.

We also found censorship worked differently on the one-on-one versus the “group” chat systems.  The latter is a WeChat feature that allows chat groups of up to 500 users.  Our tests found censorship on the group chat system was more extensive, possibly motivated by the desire to restrict speech that might mobilize large groups of people into some kind of activism.  There is also censorship of WeChat’s web browser — but, again, mostly for China-registered users.

Finally, and most troubling, we found that WeChat no longer gives a notice to users about the blocking of chat messages.  In the past, users received a warning saying they couldn’t post a message because it “contains restricted words.” Now if you send a banned keyword, it simply doesn’t appear on the recipient’s screen. It’s like it never happened at all.  This type of “silent” censorship is highly unlikely to be noticed by either communicating party unless one of them thinks to double check (or researchers like us scrutinize it closely).

By removing notice of censorship, WeChat sinks deeper into a dark hole of unaccountability to its users.

Research of this sort is essential because it helps pull back the curtain of obscurity that, unfortunately, pervades so much of our digital experiences.  As social media companies increasingly shape and control what users communicate — shape our realities — they affect our ability to exercise our rights to seek and impart information — to exercise our human rights.

China may offer the most extreme examples, as our series of reports on China-based applications has shown, but they are important to study as harbingers of a possible future.  To wit, as our report is going to publication Facebook is reportedly developing a special censorship system to comply with China’s regulations, one that would “suppress posts from appearing in users’ news feeds.”  Along with WeChat’s “One App, Two Systems” model, the services these two social media giants are offering go a long way to cementing a bifurcated, territorialized, and opaque Internet.

Read the full report here: https://citizenlab.org/2016/11/wechat-china-censorship-one-app-two-systems

What to do about “dual use” digital technologies?

The following is my written testimony to the Senate Standing Committee on Human Rights – Canada, which will take place November 30, 2016 at 11:30 AM EST and video webcast here.)*


For over a decade, the Citizen Lab at the Munk School of Global Affairs, University of Toronto has researched and documented information controls that impact the openness and security of the Internet and threaten human rights. Our mission is to produce evidence-based research on cyber security issues that are associated with human rights concerns. We study how governments and the private sector censor the Internet, social media, or mobile applications.  We have done extensive reporting on targeted digital espionage on civil society.  We have produced detailed reports on the companies that sell sophisticated spyware, networking monitoring, or other tools and document their abuse potential to raise corporate social responsibility concerns.  And we have undertaken extensive technical analysis of popular applications for hidden privacy and security risks. Our goal is to inform the public while meeting high standards of rigor through academic peer review.

Citizen Lab Research into Dual-Use Technologies

One area we are particularly concerned with is the development, sale and operation of so-called “dual-use” technologies that provide capabilities to surveil users or to censor online information at the country network level. These technologies are referred to as “dual-use” because, depending on how they are deployed, they may serve a legitimate and socially beneficial purpose, or, equally well, a purpose that undermines human rights.   

Our research on dual-use technologies has fallen into two categories — those that involve network traffic management, including deep packet inspection and content filtering, and those that involve technologies used for device intrusion for more targeted monitoring.  

The first category of our research concerns certain deep packet inspection (DPI) and Internet filtering technologies that private companies can use for traffic management, but which can also be used by Internet service providers (ISPs) to prevent entire populations from accessing politically sensitive information online and/or be used for mass surveillance. This category of research uses a combination of network measurement methods, technical interrogation tests, and other “fingerprinting” techniques to identify the presence on national networks of such technologies capable of surveillance and filtering, and, where possible, the company supplying the technology. In conducting such research, questions frequently arise regarding the corporate social responsibility practices of the companies developing and selling this technology, as several of our reports in this area have identified equipment and installations sold by companies to regimes with dubious human rights track records. Our research has spotlighted several companies — Blue Coat, Websense, Fortinet, and Netsweeper — that provide filtering and deep packet inspection systems to such rights-abusing countries.  Since Netsweeper is a Canadian headquartered company and has featured repeatedly in our research on this topic, I will provide more details about our findings with respect to them.

Netsweeper, Inc. is a privately-owned technology company based in Waterloo, Ontario, Canada, whose primary offering is an Internet content filtering product and service. The company has customers ranging from educational institutions and corporations to national-level Internet Service Providers (ISPs) and telecommunications companies. Internet filtering is widely used on institutional networks, such as schools and libraries, and networks of private companies, to restrict access to a wide range of content. However, when such filtering systems are used to implement state-mandated Internet filtering at the national level, questions around human rights — specifically access to information and freedom of expression — are implicated.

Prior research by the OpenNet Initiative (2003-2013) (an Inter-University project of which Citizen Lab was a founding partner), identified the existence of Netsweeper’s filtering technology on ISPs operating in the Middle East, including Qatar, United Arab Emirates (UAE), Yemen, and Kuwait. Working on its own, Citizen Lab subsequently outlined evidence of Netsweeper’s products on the networks of Pakistan’s leading ISP, Pakistan Telecommunication Company Limited (PTCL), in a report published in 2013, and discussed their use to block the websites of independent media, and content on religion and human rights. In 2014, we reported that Netsweeper products were being used by three ISPs based in Somalia, and raised questions about the human rights implications of selling filtering technology in a failed state. In a report on information controls in Yemen in 2015, we examined the use of Netsweeper technology to filter critical political content, independent media websites, and all URLs belonging to the Israeli (.il) top-level domain in the context of an ongoing armed conflict in which the Houthi rebels had taken over the government and the country’s main ISPs.  Most recently, we published a report on September 21, 2016 that identified Netsweeper installations on nine Bahrain-based ISPs, a country with a notoriously bad human rights record, which were being employed to block access to a range of political content.

Included in some of these reports were letters with questions that we sent to Netsweeper, which also offered to publish any response from the company in full. Aside from a defamation claim filed in January 2016, and then subsequently discontinued in its entirety on April 25, 2016, Netsweeper has not responded to us.

The second category of research where we also apply the term “dual-use” concerns the use of malicious software — “malware” — billed as a tool for “lawful intercept,” e.g. zero-day exploits and remote access trojans that enable surveillance through a user’s device.  A “zero-day” — also known as an 0day — is an undisclosed computer software vulnerability.  Zero days can be precious commodities, and are traded and sold by black, grey, and legitimate market actors.  Law enforcement and intelligence agencies purchase and use zero days or other malware — typically packaged as part of a suite of “solutions” — to surreptitiously get inside a target’s device.  When used without proper safeguards, these tools (and the services that go along with them) can lead to significant human rights abuses.

Our work in this area typically begins with a “patient zero” — someone or some organization that has been targeted with a malware-laden email or link.  In the course of the last few years, we have documented numerous cases of human rights defenders and other civil society groups being targeted with advanced commercial spyware sold by companies like Italy-based Hacking Team, UK/Germany/Swiss-based Finfisher, and Israeli-based NSO Group.  Using network scanning techniques that employ digital fingerprinting for signatures belonging to the so-called “command and control” infrastructure used by this malware, we have also been able to map the proliferation of some of these systems to a large and growing global client base, many of which are governments that have notoriously bad records concerning human rights.

The data released by Citizen Lab from these projects has inspired legal and advocacy campaigns, formed much of the evidentiary basis for measures undertaken in multiple countries to control unregulated surveillance practices (e.g., 2013 modifications to the Wassenaar Arrangement), has inspired further disclosures and investigations regarding the use of spyware and filtering technologies, and has resulted in specific remediation in the form of software updates to entire consumer populations (e.g., patches to Apple’s OSX and iOS in the case of our “Million Dollar Dissident” report).

Nonetheless, our findings are only touching on a small area of what is a very disturbing larger picture.  The market for dual-use technologies, particularly spyware, is growing rapidly. Government demand for these technologies may actually be increasing following the Snowden disclosures, which raised the bar on what is deemed de rigueur in digital surveillance, and ironically may have intensified competition around the sale of zero-day exploits, and methods for defeating increasingly pervasive end-to-end encryption and other defensive measures. For example, the U.K.’s proposed Investigatory Powers Bill, at the time of writing awaiting Royal assent before becoming law, will authorize U.K. agencies to hack into targeted devices as well as “bulk networks” — meaning all devices associated with a particular geographic area.

Although Citizen Lab research has not to date identified a Canadian-based vendor of commercial spyware selling to a rights-abusing country or being used to target human rights defenders in the course of its investigations, we know that companies selling this type of technology exist.  Furthermore, the growth of the spyware market coupled with the other circumstances outlined above, suggest it is highly likely that a Canadian vendor would at some point in the not too distant future face the choice of whether or not to sell its technology and services to a rights-abusing country — if it has not already.  Indeed, it is worth pointing out that parts of a very controversial mass surveillance system implemented in Turkey by the US-based company, Procera, were reportedly outsourced to a Canadian software development company, Northforge, after engineers at Procera threatened to resign for fear of assisting President Erdogan’s draconian policies.

What is To Be Done?

Rectifying the abuse of dual-use technologies is not a simple matter, but it is one where the Government of Canada can play a constructive role. Effective solutions that encourage respect for human rights will depend on two key components: transparency of the market, and creation of an incentive structure to which private sector actors will respond.  


The primary impediment to any progress regarding dual-use technologies of concern is the lack of transparency in the market. It is impossible for non-governmental entities to accurately gauge the scale and capabilities of the dual-use technology sector. While research such as that of the Citizen Lab and Privacy International has drawn attention to the problem and highlighted certain notorious companies, sources of research data and our capacity to undertake research are limited.  Meanwhile, new actors and technologies are regularly emerging or undergoing transformation as they change ownership, headquarters, or name. Many dual-use technology companies are not transparent about the full range of products and services they sell or their clients, and the sector as a whole is shrouded in secrecy.

With their proven potential for abuse, technologies that enable countrywide Internet filtering and digital surveillance merit increased scrutiny by the government and the public. It is telling that in many countries, government officials themselves are unable to obtain a complete picture of the technologies designed, manufactured, and serviced within their borders that could be used to suppress legitimate dissent or undermine other internationally-recognized human rights. Irrespective of whether the government chooses to regulate the sale of particular technologies, some form of mandated transparency in the market for filtering and surveillance tools is essential to addressing this information gap and informing good policy.

Mandated transparency could take a number of forms, but at a minimum will require “lawful intercept,” Internet filtering, and, possibly, DPI providers that offer their products and services in the marketplace to self-identify and report as a matter of public record. An analogous model may be found in the work of the United Nations Working Group on Mercenaries, which has drafted a proposed convention regarding regulation of private military and security companies (PMSCs). The convention envisions a general state registry of the PMSCs operating in a state’s jurisdiction, as part of a broader framework for oversight and accountability.

Transparency can emerge from research. It is noteworthy that the little we know about the abuse of dual-use technologies comes primarily from rigorous, evidence-based and interdisciplinary research of the sort Citizen Lab has done. As a complement to mandated transparency, the Government of Canada could encourage this type of mixed methods research into the dual-use technology market through research funding bodies like SSHRC and NSERC, and the Canada Research Chair program. It could also develop legislation specifically designed to provide safe harbor for security research undertaken in the public interest and incorporating responsible disclosure.

Incentivizing the Private Sector to Respect Human Rights

As the UN Guiding Principles on Business and Human Rights make clear, business enterprises have the responsibility to respect internationally-recognized human rights, in their own activities as well as activities linked to their operations, products or services. At present, however, there are few if any costs incurred by the companies that supply and service dual-use technologies when such technologies are used to violate human rights. Repeatedly we have seen that, when surveillance and filtering technologies are used against journalists, activists, and other peaceful actors, the companies involved treat the matter as “water off a duck’s back”: they assert that their products are provided for lawful purposes only, benefit society, and are beyond their control in the hands of their clients. They wait for the news cycle to pass. Many companies, particularly those that supply lawful intercept products, are further insulated by the secrecy surrounding intelligence and law enforcement work and the national security prerogatives of their clientele, most of whom lack oversight or public accountability themselves.

Yet it has become increasingly clear, as evidenced by Citizen Lab and other research, that while these technologies may be used to hunt criminals and terrorists or otherwise serve a legitimate security purpose, they are simultaneously deployed against regime critics, political opponents, and other non-violent actors with alarming frequency. Regimes that lack robust rule of law and due process while facing legitimation crises and domestic dissent simply do not distinguish among targets when leveraging the advanced technologies supplied by the private sector. It has come to light that private companies may even have detailed knowledge of attacks against civil society that are reliant on their products, as they participate in trouble-shooting delivery of malware and provide other forms of expertise to their clients. Companies, however, have managed to continue to grow and develop the sector without consequence by avoiding any form of engagement on the question of human rights.

Significant intervention is required to eliminate company expectations of immunity and prompt rights-based reform. In a forthcoming piece, my colleague Sarah McKune and I lay out several areas that we feel could help control the excesses of the commercial spyware market, by shifting the costs from the public to the spyware companies themselves, in order to generate changes in company risk-opportunity calculations, practices, and overall attitude. The drastic change in incentive structure necessary to curb the abuses of this industry will rely on a combination of (1) regulation and policy, and (2) access to remedy.

  1. Regulation and policy

Export controls are a first step in the regulatory process. The Canadian government currently has in place export controls and regulations against the sale of certain types of technologies to certain foreign jurisdictions, including those relating to “IP network communications surveillance systems or equipment” and “intrusion software” (which correspond to a large degree to the Citizen Lab research outlined above). The inclusion of these two new additions to control lists was in response to modifications made in 2013 to the Wassenaar Arrangement, of which Canada is a member. Canada has released statistics concerning 2015 export licenses including those pertaining to intrusion software and IP network surveillance, which can be found here.  Although it is impossible to know what items in particular were granted licenses or what considerations were made in doing so, it is noteworthy that within the relevant category, 2202 license applications were granted, while only 2 were denied. Regardless, export controls by themselves are insufficient to address the human rights concerns associated with these items.

As various members of the Wassenaar Arrangement rolled out implementation of the 2013 controls at the national level, the challenges of relying on export controls to address the serious rights implications of dual-use technologies became evident. One key problem is designating the scope of the items to be controlled in an appropriate and predictable manner, avoiding both over- and under-inclusion. For example, with respect to items related to “intrusion software,” certain technologies anticipated to fall within the scope of the control are also used for legitimate security research. At the same time, the 2013 controls do not cover Internet filtering and other technologies with significant human rights implications. For example, companies that provide Internet traffic management under the term “Quality of Service” (QoS) are explicitly excluded from Wassenaar targeted items. Yet, while QoS technologies are certainly integral to the proper functioning of network traffic service delivery today, they can also be used to throttle traffic or certain protocols associated with specific applications. If used in contexts where the aim is to limit free expression, privacy, or access to information — as evidenced in a rising number of troubling country cases — then human rights considerations are certainly impacted.

Lastly, the Wassenaar Arrangement’s inclusion of the 2013 controls is now on uncertain ground after the United States has given notice that it intends to renegotiate the agreement following major criticisms put forward primarily by security researchers and the private sector. The U.S. decision to reopen negotiations on these Wassenaar controls will, in turn, almost certainly affect Canada’s obligations.  

A second challenge lies in the export licensing process carried out at the national level. Even when a dual-use technology is subject to control, the licensing process must be properly calibrated to address the end users and end uses of concern from a human rights perspective. This accounting requires an ever-evolving assessment, combined with the political will to both curb access within a broad group of countries (some of which may be of strategic importance to Canada) and restrict the sales of domestic corporations. As we have witnessed, the post-2013 licensing processes surrounding spyware have left much to be desired: Italian authorities’ approved an initial grant of a “global authorization” to Hacking Team, which permitted it to export its spyware to destinations such as Kazakhstan; and, the Israeli authorities gave approval to NSO Group to export sophisticated iOS zero-day exploits to the United Arab Emirates, where we discovered they were subsequently used against a peaceful dissident and other political targets.

For these and other reasons, export controls, while important, constitute only one means by which the Government of Canada can help constrain the abuse of dual-use technologies. In tailoring applicable export controls, Canada can certainly take a proactive stance on addressing the end users and end uses that pose human rights risks. At the same time, however, such efforts can be complemented by additional regulatory and policy measures. Measures worth exploring include:

  • Government procurement and export credit or assistance policies that require vendors of dual-use technologies to demonstrate company commitment to and record of human rights due diligence. Vendors that have engaged in fraudulent or illegal practices, or have supplied technology that has facilitated human rights abuses, should be ineligible for award of government contracts or support in any form.
  • Enhanced consumer protection laws and active efforts at consumer protection agencies to address the misuse of DPI, Internet filtering technology, and spyware against the public.
  • A regulatory framework for oversight and accountability specifically tailored to dual-use technologies. That proposed in the context of PMSCs, as noted above, offers a number of elements that could be considered for inclusion, such as enumerating prohibited activities; establishing requirements for training of personnel; assessing company compliance with domestic and international law; and investigating reports of violations.
  • Structured dialogue with companies and civil society regarding the establishment of industry self-regulation, which can be modeled on the International Code of Conduct for Private Security Service Providers and its multistakeholder association. Such a dialogue could include work on model contracts and best practices for “lawful intercept” and Internet filtering technology providers.

(2) Access to remedy

When dual-use technology companies provide products and services used to undermine human rights, or when they engage in practices that are fraudulent or illegal in relevant jurisdictions (e.g., practices that are violative of intellectual property, consumer protection, privacy, or computer crime laws), it is appropriate that those harmed by such activity may seek remedy against them. Canadian law could ensure that criminal or civil litigation is possible in such circumstances, including through the clear establishment of jurisdiction over actors that operate transnationally or may be state-linked. Exposure to liability for misconduct will be the primary motivating force behind any change in this sector.

The Government of Canada is a vocal supporter of Internet freedom and human rights, and is a member in all of the relevant international bodies in which such topics are discussed.

But the fact that Citizen Lab has documented at least seven countries whose national ISPs use or have used a Canadian company’s services to censor Internet content protected under internationally-recognized human rights agreements is an embarrassing black mark for all Canadians. While we have no evidence that a Canadian intrusion software, DPI, or IP monitoring vendor has sold its services to a rights-abusing country that does not necessarily mean it has not happened, or will not happen in the future.  The Turkey-Procera case, outlined earlier, should certainly raise alarm bells.

By proactively addressing the regulation of dual-use technologies in ways outlined above, the Government of Canada would align its actions with its words, and ensure business considerations are not undertaken without human rights concerns being addressed.

*The author gratefully acknowledges the input of Sarah McKune, Senior Legal Advisor, Citizen Lab, who assisted in the preparation and writing of this testimony and John Scott Railton, Citizen Lab senior researcher, for comments and feedback.

Just Enough to Do the Job: Targeted Attacks on Tibetans

I am pleased to announce a new Citizen Lab report, entitled “It’s Parliamentary: KeyBoy and the targeting of the Tibetan Community.” The report is authored by the Citizen Lab’s Adam Hulcoop, Etienne Maynier, John Scott Railton, Masashi Crete-Nishihata, and Matt Brooks and can be found here: https://citizenlab.org/2016/11/parliament-keyboy/

In this report, the authors track a malware operation targeting members of the Tibetan Parliament over August and October 2016.  The operations involved highly targeted email lures with repurposed content and attachments that contained an updated version of custom backdoor known as “KeyBoy.”

There are several noteworthy parts of this report:

First, this operation is another example of a threat actor using “just enough” technical sophistication to exploit a target.  A significant amount of resources go into a targeted espionage operation, from crafting of an exploit to its packaging and delivery to the intended target, to the command and control infrastructure, and more.  From the perspective of an operator, why risk burning some of these precious resources when something less sophisticated will do? Throughout the many years we have been studying targeted digital attacks on the Tibetan community, we have seen operators using the same old patched exploits because … well, they work.

Part of the reason these attacks work is that the communities in question typically do not have the resources or capabilities to protect their networks properly.  While the Tibetan diaspora has done a remarkable job educating their community about how to recognize a suspicious email or attachment and not open it (their Detach from Attachments campaign being one example) many of them are still reliant on un-patched operating systems and a lack of adequate digital security controls.  As Citizen Lab’s Adam Hulcoop remarked, “We found it striking that the operators made only the bare minimum technical changes to avoid antivirus detection, while sticking with ‘old day’ exploits that would not work on a patched and updated system.”

What goes for Tibetans holds true across the entire civil society space: NGOs are typically small, overstretched organizations; most have few resources to dedicate to doing digital security well.  As a consequence, operators of targeted espionage campaigns can hold their big weapons in reserve, put more of their effort into crafting enticing messages — into the social engineering part of the equation — while re-purposing older exploits like KeyBoy.  As Citizen Lab senior researcher John Scott Railton notes in a recent article, “Citizen Lab research has consistently found that although the overall technical sophistication of attacks is typically low, their social engineering sophistication is much higher.”  The reason is that civil society is “chronically under-resourced, often relying on unmanaged networks and endpoints, combined with extensive use of popular online platforms….[providing] a target-rich environment for attackers.”

The second noteworthy part of the report concerns the precision around the social engineering on the part of the operators. The attacks were remarkably well timed to maximize return on victims.  Just 15 hours after members of the Tibetan parliament received an email about an upcoming conference, they received another email with the same subject and attachment, this time crafted to exploit a vulnerability in Microsoft Office using KeyBoy.  This level of targeting and re-use of a legitimate document sent only hours before shows how closely the Tibetans are watched by their adversaries, and how much effort the operators of such attacks put into the social engineering part of the targeted attack.  With such persistence and craftiness on the part of threat operators, it is no wonder civil society groups are facing an epidemic of these type of campaigns.

Finally, the report demonstrates the value of trusted partnerships with targeted communities.  The Citizen Lab has worked with Tibetan communities for nearly a decade, and during that time we have learned a great deal from each other.  That they are willing to share samples of attacks like these with our researchers shows not only their determination to better protect themselves, but a recognition of the value of careful evidence-based research for their community.  By publishing this report, we hope that civil society, human rights defenders, and their sponsors and supporters can better understand the threat environment, and take steps to protect themselves.  

To that end, alongside the report, we are publishing extensive details and indicators of compromise in several appendices to the report, and hope other researchers will continue where we left off.

Read the report here: https://citizenlab.org/2016/11/parliament-keyboy/

What Lies Beneath China’s Live-Streaming Apps?

Today, the Citizen Lab is releasing a new report, entitled: “Harmonized Histories? A year of fragmented censorship across Chinese live streaming platforms.”  The report is part of our NetAlert series, and can be found here.

Live-streaming media apps are extraordinarily popular in mainland China, used by millions.  Similar in functionality to the US-based and Twitter-owned streaming media app, Periscope (which is banned in China) China-based apps like YY, 9158, and Sina Show, have become a major Internet craze.  Users of these apps share everything from karaoke and live poker matches to pop culture commentary and voyeuristic peeks into their private lives.  For example, Zhou Xiaohu, a 30-year-old construction worker from Inner Mongolia, films himself eating dinner and watching TV, while another live-streamer earns thousands of yuan taking viewers on tours of Japan’s red-light districts.

The apps are also big business opportunities, for both users and the companies that operate them.  Popular streamers receive virtual gifts from their fans, who can number in the hundreds of thousands for some of the most widely viewed. The streamers can exchange these virtual gifts for cash.  Some of them have become millionaires as a result. The platforms themselves are also highly lucrative, attracting venture capital and advertisement revenues.

Chinese authorities have taken notice of the exploding live-streaming universe, which is not surprising considering their strict concerns over free expression.  Occasionally streams will veer into taboo topics, such as politics or pornography, which has resulted in more scrutiny, fines, takedowns, and increased censorship.

To better understand how censorship on the platforms takes place, our researchers downloaded three of the most popular applications (YY, 9158, and Sina Show) and systematically reverse engineered them.   Doing so allowed us to extract the banned keywords hidden in the clients as they are regularly updated.  Between February 2015 and October 2016, we collected 19,464 unique keywords that triggered censorship on the chats associated with each application, which we then translated, analyzed, and categorized.

What we found is interesting for several reasons, and runs counter to claims put forth in a widely-read study on China’s Internet censorship system authored by Gary King et al and published in the American Political Science Review.  In that study, King and his colleagues conclude that China’s censors are not concerned with “posts with negative, even vitriolic, criticism of the state, its leaders, and its policies” and instead focus predominantly on “curtailing collective action by silencing comments that represent, reinforce, or spur social mobilization, regardless of content.”  Their analysis gives the impression of a centralized and monolithic censorship system to which all Internet providers and companies strictly conform.

We found, on the other hand, that there is significant variation in blocking across the platforms.  This variation means that while the Chinese authorities may set general expectations of taboo or controversial topics to be avoided, what, exactly, to filter is left to the discretion of the companies themselves to implement.

We also found, contrary of King et al, that content they suggested was tolerated was actually routinely censored by the live-streaming companies, albeit in inconsistent ways across each of the platforms.  We also found all sorts of keywords targeted for filtering that had nothing to do with political directives, including censoring of posts by live-streaming applications related to their business competitors.

In other words, our research shows that the social media ecosystem in China — though definitely restricted for users — is more decentralized, variable, and chaotic than what King and his colleagues claim. It confirms the role of intermediary liability in China that Rebecca Mackinnon has put forward, known as “self discipline,” whereby companies are expected to police themselves and their users to ensure a “harmonious and healthy Internet.”  Ironically, that self-discipline often results in entirely different implementations of censorship on individual platforms, and a less than “harmonious” Internet experience as a result.

Our reverse engineering also discovered that YY — the most popular of the live-streaming apps, with over 844 million registered users — undertakes surveillance of users’ chats. When a censored keyword is entered by a user, a message is sent back to YY’s servers that includes the username of who sent the message, the username of who received the message, the keyword that triggered censorship, and the entire triggering message. Nearly a billion unwitting users’ chats subject to hidden keyword surveillance!  Recall that in China companies are required to share user information with security agencies upon request, and Chinese citizens have been arrested based entirely on their online actions.  Recently, for example, one user posted an image of a police report of a person under investigation for downloading a VPN on his or her mobile phone.

On a more technical level, our research shows the value of careful reverse engineering for revealing information controls hidden from the view of the typical user.  The keyword lists we extracted and are publishing reveal exactly what content triggers censorship and surveillance, something that is known only to the decision makers within the companies themselves.  We see this type of research as critical to informing users of the ecosystem within which they communicate.

Sometimes what we find also runs counter to conventional wisdom.  You don’t know what’s being censored if you can’t see the list of banned keywords. Opening these applications up allows us to see them from the inside-out in a truly unbiased way that other more impressionistic scans can only infer.

What an “MRI of the Internet” Can Reveal: Netsweeper in Bahrain

I am pleased to announce a new Citizen Lab report: “Tender Confirmed, Rights At Risk: Verifying Netsweeper in Bahrain.”  The full report can be found here: https://citizenlab.org/2016/09/tender-confirmed-rights-risk-verifying-netsweeper-bahrain

Internet censorship is a major and growing human rights issue today. Access to content is restricted for users on social media, like Facebook, on mobile applications, and on search engines.  The most egregious form of censorship, however, is that which occurs at a national level for entire populations.  This type of censorship has been spreading for many years, and now has become normalized across numerous countries.

One of the Citizen Lab’s longest standing forms of research is the meticulous documentation of Internet censorship.  We were one of the founding partners of the OpenNet Initiative, which at one time documented Internet filtering and surveillance in more than 70 countries on an annual basis. We continue this research in the form of country case studies or analyses of information controls around specific events, like a civil war.

At the core of this research is the use of a mixture of technical interrogation and network measurements methods, including in-country testing, remote scans of national networks, queries on databases, and large-area scans of the entire Internet.  One of the methods we use in this research is a scanning tool called Zmap, which we run on high-speed computers to perform a complete scan of the entire Internet space in a matter of minutes.  Think of this technique as an MRI of the Internet.

A byproduct of these scans is the ability to identify equipment that is used to undertake Internet censorship and surveillance. Certain filtering systems have the equivalent of digital signatures which we use when scanning the Internet. Searching for these signatures allows us to locate installations around the world. Doing so is useful in and of itself to help shed a light on what’s going on beneath the surface of the Internet. But it is also useful for raising awareness about the companies that are complicit in Internet censorship practices.

One of the companies that we have identified in this way is Netsweeper, Inc, a Canadian company based in Waterloo, Ontario. We have identified Netsweeper installations being used to filter at the national level in Pakistan, Somalia, and Yemen, among others.  Our latest report, published today, locates live Netsweeper installations on nine ISPs in the Kingdom of Bahrain.

These findings are significant for several reasons: Bahrain is one of the world’s worst countries for respect for human rights, particularly press and Internet freedoms.  For many years, Bahrain has restricted access to Internet content having to do with independent media, websites critical of the Kingdom, and content related to the Shia faith, which is heavily persecuted in Bahrain.

In January 2016, Bahrain issued a tender for bidding on a national-level Internet filtering system. Our findings are significant because we can confirm the presence of Netsweeper installations on Bahraini ISPs following the bid.

These findings are also noteworthy because Netsweeper filed, and then discontinued a $3.5 million defamation suit against myself and the University of Toronto following our prior report on Netsweeper in Yemen.   Our report published today is the first since the defamation suit was discontinued by Netsweeper. As we have done with prior reports, we sent Netsweeper a letter, which can be found here, in which we lay out our findings, ask Netsweeper questions about their due diligence and corporate social responsibility policies, and offer to publish their response in full alongside our report. As of today, Netsweeper has not replied to that letter.

Lastly, the case is significant because Netsweeper is a Canadian company, and the provision of Internet filtering services to a country like Bahrain— though not in violation of any Canadian law per se — is definitely being used to suppress content deemed legitimate expression under international human rights law, which Canada explicitly supports.  All the more troubling, then, is the fact that Netsweeper has benefited, and will benefit in the future, from tangible support provided by both the Canadian and the Ontario governments in trade shows held in the Gulf region.  Canada’s Trade Commissioner says the government’s involvement at these trade shows includes assistance with “business-to-business meetings” and “networking events” as well as provision of a “pavilion/exhibit” — all of which is “offered free of charge to Canadian companies and organizations.”  While we have no evidence Canada went so far as to facilitate Netsweeper’s specific bid on Bahrain’s tender, they certainly did use Canadian tax payers dollars to represent Netsweeper to interested clients in the region.

Should the government of Canada be promoting a company whose software is used to violate human rights and which offers services in direct contradiction to our stated foreign policy goals on cyberspace?   Perhaps a more harmonized approach would be to require companies like Netsweeper to have some explicit corporate social responsibility process in place.  Export controls could be established that restrict the sale of technology and services to countries that will use their product to infringe internationally-recognized human rights.  Taking these steps would help better synchronize Canada’s economic and human rights policies while also bringing the world of Internet filtering in line with widely recognized principles on how businesses should respect human rights.