Days are long gone when we used to interact with the Internet as an undifferentiated network. The reality today is that what we communicate online is mediated by companies that own and operate the Internet services we use.  Social media in particular have become, for an increasing number of people, their windows on reality.  Whether, and in what ways, those windows might be distorted — by corporate practices or government directives — is thus a matter of significant public importance (but not always easy to discern with the naked eye).

Take the case of WeChat — the most popular chat application in China, and the fourth largest in the world with 806 million monthly active users.  WeChat is more than just an instant messaging application. It is more like a lifestyle platform.  WeChat subscribers use the app not only to send text, voice, and video but to play games, make mobile payments, hail taxis, and more.

As with all other Internet services operating in China, however, WeChat must comply with extensive government regulations that require companies to police their networks and users, and share user data with security agencies upon request.  Over numerous recent case-study reports, Citizen Lab research has found that many China-based applications follow these regulations by building into their applications hidden keyword censorship and surveillance.  WeChat is no exception, although with a twist.

Today, we are releasing a new report, entitled “One App, Two Systems: How WeChat uses one censorship policy in China and another internationally.”  For this report, we undertook several controlled experiments using combinations of China, Canada, and U.S. registered phone numbers and accounts to test for Internet censorship on WeChat’s platform.  What we found was quite surprising.

Turns out that there is substantial censorship on WeChat, but split along several dimensions.  There is keyword filtering for users registered with a mainland China phone number but not for those registering with an international number.  However, we also found that once a China-based user had registered with a mainland China phone number, the censorship follows them around — even if they switch to an international phone number, or work, travel, or study abroad.  To give some context, there are roughly 50 million overseas Chinese people working and living abroad.  China’s “One-App, Two Systems” keeps them under the control of China’s censorship regime no matter where they go. This extra-territorial application of information controls is quite unique, and certainly a disturbing precedent to set.

We also found censorship worked differently on the one-on-one versus the “group” chat systems.  The latter is a WeChat feature that allows chat groups of up to 500 users.  Our tests found censorship on the group chat system was more extensive, possibly motivated by the desire to restrict speech that might mobilize large groups of people into some kind of activism.  There is also censorship of WeChat’s web browser — but, again, mostly for China-registered users.

Finally, and most troubling, we found that WeChat no longer gives a notice to users about the blocking of chat messages.  In the past, users received a warning saying they couldn’t post a message because it “contains restricted words.” Now if you send a banned keyword, it simply doesn’t appear on the recipient’s screen. It’s like it never happened at all.  This type of “silent” censorship is highly unlikely to be noticed by either communicating party unless one of them thinks to double check (or researchers like us scrutinize it closely).

By removing notice of censorship, WeChat sinks deeper into a dark hole of unaccountability to its users.

Research of this sort is essential because it helps pull back the curtain of obscurity that, unfortunately, pervades so much of our digital experiences.  As social media companies increasingly shape and control what users communicate — shape our realities — they affect our ability to exercise our rights to seek and impart information — to exercise our human rights.

China may offer the most extreme examples, as our series of reports on China-based applications has shown, but they are important to study as harbingers of a possible future.  To wit, as our report is going to publication Facebook is reportedly developing a special censorship system to comply with China’s regulations, one that would “suppress posts from appearing in users’ news feeds.”  Along with WeChat’s “One App, Two Systems” model, the services these two social media giants are offering go a long way to cementing a bifurcated, territorialized, and opaque Internet.

Read the full report here: https://citizenlab.org/2016/11/wechat-china-censorship-one-app-two-systems

The following is my written testimony to the Senate Standing Committee on Human Rights – Canada, which will take place November 30, 2016 at 11:30 AM EST and video webcast here.)*

Background

For over a decade, the Citizen Lab at the Munk School of Global Affairs, University of Toronto has researched and documented information controls that impact the openness and security of the Internet and threaten human rights. Our mission is to produce evidence-based research on cyber security issues that are associated with human rights concerns. We study how governments and the private sector censor the Internet, social media, or mobile applications.  We have done extensive reporting on targeted digital espionage on civil society.  We have produced detailed reports on the companies that sell sophisticated spyware, networking monitoring, or other tools and document their abuse potential to raise corporate social responsibility concerns.  And we have undertaken extensive technical analysis of popular applications for hidden privacy and security risks. Our goal is to inform the public while meeting high standards of rigor through academic peer review.

Citizen Lab Research into Dual-Use Technologies

One area we are particularly concerned with is the development, sale and operation of so-called “dual-use” technologies that provide capabilities to surveil users or to censor online information at the country network level. These technologies are referred to as “dual-use” because, depending on how they are deployed, they may serve a legitimate and socially beneficial purpose, or, equally well, a purpose that undermines human rights.   

Our research on dual-use technologies has fallen into two categories — those that involve network traffic management, including deep packet inspection and content filtering, and those that involve technologies used for device intrusion for more targeted monitoring.  

The first category of our research concerns certain deep packet inspection (DPI) and Internet filtering technologies that private companies can use for traffic management, but which can also be used by Internet service providers (ISPs) to prevent entire populations from accessing politically sensitive information online and/or be used for mass surveillance. This category of research uses a combination of network measurement methods, technical interrogation tests, and other “fingerprinting” techniques to identify the presence on national networks of such technologies capable of surveillance and filtering, and, where possible, the company supplying the technology. In conducting such research, questions frequently arise regarding the corporate social responsibility practices of the companies developing and selling this technology, as several of our reports in this area have identified equipment and installations sold by companies to regimes with dubious human rights track records. Our research has spotlighted several companies — Blue Coat, Websense, Fortinet, and Netsweeper — that provide filtering and deep packet inspection systems to such rights-abusing countries.  Since Netsweeper is a Canadian headquartered company and has featured repeatedly in our research on this topic, I will provide more details about our findings with respect to them.

Netsweeper, Inc. is a privately-owned technology company based in Waterloo, Ontario, Canada, whose primary offering is an Internet content filtering product and service. The company has customers ranging from educational institutions and corporations to national-level Internet Service Providers (ISPs) and telecommunications companies. Internet filtering is widely used on institutional networks, such as schools and libraries, and networks of private companies, to restrict access to a wide range of content. However, when such filtering systems are used to implement state-mandated Internet filtering at the national level, questions around human rights — specifically access to information and freedom of expression — are implicated.

Prior research by the OpenNet Initiative (2003-2013) (an Inter-University project of which Citizen Lab was a founding partner), identified the existence of Netsweeper’s filtering technology on ISPs operating in the Middle East, including Qatar, United Arab Emirates (UAE), Yemen, and Kuwait. Working on its own, Citizen Lab subsequently outlined evidence of Netsweeper’s products on the networks of Pakistan’s leading ISP, Pakistan Telecommunication Company Limited (PTCL), in a report published in 2013, and discussed their use to block the websites of independent media, and content on religion and human rights. In 2014, we reported that Netsweeper products were being used by three ISPs based in Somalia, and raised questions about the human rights implications of selling filtering technology in a failed state. In a report on information controls in Yemen in 2015, we examined the use of Netsweeper technology to filter critical political content, independent media websites, and all URLs belonging to the Israeli (.il) top-level domain in the context of an ongoing armed conflict in which the Houthi rebels had taken over the government and the country’s main ISPs.  Most recently, we published a report on September 21, 2016 that identified Netsweeper installations on nine Bahrain-based ISPs, a country with a notoriously bad human rights record, which were being employed to block access to a range of political content.

Included in some of these reports were letters with questions that we sent to Netsweeper, which also offered to publish any response from the company in full. Aside from a defamation claim filed in January 2016, and then subsequently discontinued in its entirety on April 25, 2016, Netsweeper has not responded to us.

The second category of research where we also apply the term “dual-use” concerns the use of malicious software — “malware” — billed as a tool for “lawful intercept,” e.g. zero-day exploits and remote access trojans that enable surveillance through a user’s device.  A “zero-day” — also known as an 0day — is an undisclosed computer software vulnerability.  Zero days can be precious commodities, and are traded and sold by black, grey, and legitimate market actors.  Law enforcement and intelligence agencies purchase and use zero days or other malware — typically packaged as part of a suite of “solutions” — to surreptitiously get inside a target’s device.  When used without proper safeguards, these tools (and the services that go along with them) can lead to significant human rights abuses.

Our work in this area typically begins with a “patient zero” — someone or some organization that has been targeted with a malware-laden email or link.  In the course of the last few years, we have documented numerous cases of human rights defenders and other civil society groups being targeted with advanced commercial spyware sold by companies like Italy-based Hacking Team, UK/Germany/Swiss-based Finfisher, and Israeli-based NSO Group.  Using network scanning techniques that employ digital fingerprinting for signatures belonging to the so-called “command and control” infrastructure used by this malware, we have also been able to map the proliferation of some of these systems to a large and growing global client base, many of which are governments that have notoriously bad records concerning human rights.

The data released by Citizen Lab from these projects has inspired legal and advocacy campaigns, formed much of the evidentiary basis for measures undertaken in multiple countries to control unregulated surveillance practices (e.g., 2013 modifications to the Wassenaar Arrangement), has inspired further disclosures and investigations regarding the use of spyware and filtering technologies, and has resulted in specific remediation in the form of software updates to entire consumer populations (e.g., patches to Apple’s OSX and iOS in the case of our “Million Dollar Dissident” report).

Nonetheless, our findings are only touching on a small area of what is a very disturbing larger picture.  The market for dual-use technologies, particularly spyware, is growing rapidly. Government demand for these technologies may actually be increasing following the Snowden disclosures, which raised the bar on what is deemed de rigueur in digital surveillance, and ironically may have intensified competition around the sale of zero-day exploits, and methods for defeating increasingly pervasive end-to-end encryption and other defensive measures. For example, the U.K.’s proposed Investigatory Powers Bill, at the time of writing awaiting Royal assent before becoming law, will authorize U.K. agencies to hack into targeted devices as well as “bulk networks” — meaning all devices associated with a particular geographic area.

Although Citizen Lab research has not to date identified a Canadian-based vendor of commercial spyware selling to a rights-abusing country or being used to target human rights defenders in the course of its investigations, we know that companies selling this type of technology exist.  Furthermore, the growth of the spyware market coupled with the other circumstances outlined above, suggest it is highly likely that a Canadian vendor would at some point in the not too distant future face the choice of whether or not to sell its technology and services to a rights-abusing country — if it has not already.  Indeed, it is worth pointing out that parts of a very controversial mass surveillance system implemented in Turkey by the US-based company, Procera, were reportedly outsourced to a Canadian software development company, Northforge, after engineers at Procera threatened to resign for fear of assisting President Erdogan’s draconian policies.

What is To Be Done?

Rectifying the abuse of dual-use technologies is not a simple matter, but it is one where the Government of Canada can play a constructive role. Effective solutions that encourage respect for human rights will depend on two key components: transparency of the market, and creation of an incentive structure to which private sector actors will respond.  

Transparency

The primary impediment to any progress regarding dual-use technologies of concern is the lack of transparency in the market. It is impossible for non-governmental entities to accurately gauge the scale and capabilities of the dual-use technology sector. While research such as that of the Citizen Lab and Privacy International has drawn attention to the problem and highlighted certain notorious companies, sources of research data and our capacity to undertake research are limited.  Meanwhile, new actors and technologies are regularly emerging or undergoing transformation as they change ownership, headquarters, or name. Many dual-use technology companies are not transparent about the full range of products and services they sell or their clients, and the sector as a whole is shrouded in secrecy.

With their proven potential for abuse, technologies that enable countrywide Internet filtering and digital surveillance merit increased scrutiny by the government and the public. It is telling that in many countries, government officials themselves are unable to obtain a complete picture of the technologies designed, manufactured, and serviced within their borders that could be used to suppress legitimate dissent or undermine other internationally-recognized human rights. Irrespective of whether the government chooses to regulate the sale of particular technologies, some form of mandated transparency in the market for filtering and surveillance tools is essential to addressing this information gap and informing good policy.

Mandated transparency could take a number of forms, but at a minimum will require “lawful intercept,” Internet filtering, and, possibly, DPI providers that offer their products and services in the marketplace to self-identify and report as a matter of public record. An analogous model may be found in the work of the United Nations Working Group on Mercenaries, which has drafted a proposed convention regarding regulation of private military and security companies (PMSCs). The convention envisions a general state registry of the PMSCs operating in a state’s jurisdiction, as part of a broader framework for oversight and accountability.

Transparency can emerge from research. It is noteworthy that the little we know about the abuse of dual-use technologies comes primarily from rigorous, evidence-based and interdisciplinary research of the sort Citizen Lab has done. As a complement to mandated transparency, the Government of Canada could encourage this type of mixed methods research into the dual-use technology market through research funding bodies like SSHRC and NSERC, and the Canada Research Chair program. It could also develop legislation specifically designed to provide safe harbor for security research undertaken in the public interest and incorporating responsible disclosure.

Incentivizing the Private Sector to Respect Human Rights

As the UN Guiding Principles on Business and Human Rights make clear, business enterprises have the responsibility to respect internationally-recognized human rights, in their own activities as well as activities linked to their operations, products or services. At present, however, there are few if any costs incurred by the companies that supply and service dual-use technologies when such technologies are used to violate human rights. Repeatedly we have seen that, when surveillance and filtering technologies are used against journalists, activists, and other peaceful actors, the companies involved treat the matter as “water off a duck’s back”: they assert that their products are provided for lawful purposes only, benefit society, and are beyond their control in the hands of their clients. They wait for the news cycle to pass. Many companies, particularly those that supply lawful intercept products, are further insulated by the secrecy surrounding intelligence and law enforcement work and the national security prerogatives of their clientele, most of whom lack oversight or public accountability themselves.

Yet it has become increasingly clear, as evidenced by Citizen Lab and other research, that while these technologies may be used to hunt criminals and terrorists or otherwise serve a legitimate security purpose, they are simultaneously deployed against regime critics, political opponents, and other non-violent actors with alarming frequency. Regimes that lack robust rule of law and due process while facing legitimation crises and domestic dissent simply do not distinguish among targets when leveraging the advanced technologies supplied by the private sector. It has come to light that private companies may even have detailed knowledge of attacks against civil society that are reliant on their products, as they participate in trouble-shooting delivery of malware and provide other forms of expertise to their clients. Companies, however, have managed to continue to grow and develop the sector without consequence by avoiding any form of engagement on the question of human rights.

Significant intervention is required to eliminate company expectations of immunity and prompt rights-based reform. In a forthcoming piece, my colleague Sarah McKune and I lay out several areas that we feel could help control the excesses of the commercial spyware market, by shifting the costs from the public to the spyware companies themselves, in order to generate changes in company risk-opportunity calculations, practices, and overall attitude. The drastic change in incentive structure necessary to curb the abuses of this industry will rely on a combination of (1) regulation and policy, and (2) access to remedy.

1. Regulation and policy

Export controls are a first step in the regulatory process. The Canadian government currently has in place export controls and regulations against the sale of certain types of technologies to certain foreign jurisdictions, including those relating to “IP network communications surveillance systems or equipment” and “intrusion software” (which correspond to a large degree to the Citizen Lab research outlined above). The inclusion of these two new additions to control lists was in response to modifications made in 2013 to the Wassenaar Arrangement, of which Canada is a member. Canada has released statistics concerning 2015 export licenses including those pertaining to intrusion software and IP network surveillance, which can be found here.  Although it is impossible to know what items in particular were granted licenses or what considerations were made in doing so, it is noteworthy that within the relevant category, 2202 license applications were granted, while only 2 were denied. Regardless, export controls by themselves are insufficient to address the human rights concerns associated with these items.

As various members of the Wassenaar Arrangement rolled out implementation of the 2013 controls at the national level, the challenges of relying on export controls to address the serious rights implications of dual-use technologies became evident. One key problem is designating the scope of the items to be controlled in an appropriate and predictable manner, avoiding both over- and under-inclusion. For example, with respect to items related to “intrusion software,” certain technologies anticipated to fall within the scope of the control are also used for legitimate security research. At the same time, the 2013 controls do not cover Internet filtering and other technologies with significant human rights implications. For example, companies that provide Internet traffic management under the term “Quality of Service” (QoS) are explicitly excluded from Wassenaar targeted items. Yet, while QoS technologies are certainly integral to the proper functioning of network traffic service delivery today, they can also be used to throttle traffic or certain protocols associated with specific applications. If used in contexts where the aim is to limit free expression, privacy, or access to information — as evidenced in a rising number of troubling country cases — then human rights considerations are certainly impacted.

Lastly, the Wassenaar Arrangement’s inclusion of the 2013 controls is now on uncertain ground after the United States has given notice that it intends to renegotiate the agreement following major criticisms put forward primarily by security researchers and the private sector. The U.S. decision to reopen negotiations on these Wassenaar controls will, in turn, almost certainly affect Canada’s obligations.  

A second challenge lies in the export licensing process carried out at the national level. Even when a dual-use technology is subject to control, the licensing process must be properly calibrated to address the end users and end uses of concern from a human rights perspective. This accounting requires an ever-evolving assessment, combined with the political will to both curb access within a broad group of countries (some of which may be of strategic importance to Canada) and restrict the sales of domestic corporations. As we have witnessed, the post-2013 licensing processes surrounding spyware have left much to be desired: Italian authorities’ approved an initial grant of a “global authorization” to Hacking Team, which permitted it to export its spyware to destinations such as Kazakhstan; and, the Israeli authorities gave approval to NSO Group to export sophisticated iOS zero-day exploits to the United Arab Emirates, where we discovered they were subsequently used against a peaceful dissident and other political targets.

For these and other reasons, export controls, while important, constitute only one means by which the Government of Canada can help constrain the abuse of dual-use technologies. In tailoring applicable export controls, Canada can certainly take a proactive stance on addressing the end users and end uses that pose human rights risks. At the same time, however, such efforts can be complemented by additional regulatory and policy measures. Measures worth exploring include:

• Government procurement and export credit or assistance policies that require vendors of dual-use technologies to demonstrate company commitment to and record of human rights due diligence. Vendors that have engaged in fraudulent or illegal practices, or have supplied technology that has facilitated human rights abuses, should be ineligible for award of government contracts or support in any form.
• Enhanced consumer protection laws and active efforts at consumer protection agencies to address the misuse of DPI, Internet filtering technology, and spyware against the public.
• A regulatory framework for oversight and accountability specifically tailored to dual-use technologies. That proposed in the context of PMSCs, as noted above, offers a number of elements that could be considered for inclusion, such as enumerating prohibited activities; establishing requirements for training of personnel; assessing company compliance with domestic and international law; and investigating reports of violations.
• Structured dialogue with companies and civil society regarding the establishment of industry self-regulation, which can be modeled on the International Code of Conduct for Private Security Service Providers and its multistakeholder association. Such a dialogue could include work on model contracts and best practices for “lawful intercept” and Internet filtering technology providers.

(2) Access to remedy

When dual-use technology companies provide products and services used to undermine human rights, or when they engage in practices that are fraudulent or illegal in relevant jurisdictions (e.g., practices that are violative of intellectual property, consumer protection, privacy, or computer crime laws), it is appropriate that those harmed by such activity may seek remedy against them. Canadian law could ensure that criminal or civil litigation is possible in such circumstances, including through the clear establishment of jurisdiction over actors that operate transnationally or may be state-linked. Exposure to liability for misconduct will be the primary motivating force behind any change in this sector.

The Government of Canada is a vocal supporter of Internet freedom and human rights, and is a member in all of the relevant international bodies in which such topics are discussed.

But the fact that Citizen Lab has documented at least seven countries whose national ISPs use or have used a Canadian company’s services to censor Internet content protected under internationally-recognized human rights agreements is an embarrassing black mark for all Canadians. While we have no evidence that a Canadian intrusion software, DPI, or IP monitoring vendor has sold its services to a rights-abusing country that does not necessarily mean it has not happened, or will not happen in the future.  The Turkey-Procera case, outlined earlier, should certainly raise alarm bells.

By proactively addressing the regulation of dual-use technologies in ways outlined above, the Government of Canada would align its actions with its words, and ensure business considerations are not undertaken without human rights concerns being addressed.

*The author gratefully acknowledges the input of Sarah McKune, Senior Legal Advisor, Citizen Lab, who assisted in the preparation and writing of this testimony and John Scott Railton, Citizen Lab senior researcher, for comments and feedback.