Internet outage in China on Jan 21

Yesterday we witnessed one of the largest Internet outages ever in China. We have three theories about why this outage may have occurred - two related to the Falun Gong but our third theory is that the Chinese authorities set out to attack our unblockable mirror websites.

From 15:30 to 16:30 (China time) on January 21, DNS lookup to any domain would incorrectly resolve to 65.49.2.178. Websites inside and outside of China were affected. Even Baidu and Sina were inaccessible. Only software using IP directly (e.g. QQ, VPNs) worked during that time. Attempts to visit any website redirected to http://65.49.2.178, which didn’t respond during that time.  The overwhelming traffic to this IP likely crashed the server.

Timeline

Event

15:15

GFW DNS poisoning begins. First recorded instance.

15:17

Local DNS servers began to cache incorrect responses. Some large websites in China began to be affected e.g Sina Weibo.

 

Incorrect DNS continue to spread through Chinese DNS servers. Major websites including Baidu, Sina affected.

15:39

DNS poisoning lifted by GFW. But local DNS resolvers cached incorrect responses. Users continued to experience outage.

16:00

ISPs around China were manually flushing DNS caches and connections were gradually restored.

We have conclusive evidence that this outage was caused by the Great Firewall (GFW). DNS poisoning is used extensively by the GFW. Some articles that have appeared about this outage suspected that the root DNS server in China was hacked and all domains hijacked to 65.49.2.178. This could explain why DNS servers in China were poisoned. However, during that time, we see that a lookup to 8.8.8.8, a public DNS operated by Google, returned bogus results if the lookup was done from China. In fact, the Google public DNS was not poisoned; the bogus response 65.49.2.178 could only have been returned by GFW.  If the Chinese root DNS server was hacked, a DNS lookup in China via 8.8.8.8 should have returned a correct response. See the below image from Zhihu.

Our testing system is designed to detect these bogus responses by querying non-existent DNS servers outside of China. Any valid response must come via GFW. We indeed observed such behavior during that time on all domains.  

But why did GFW poison all domains and effectively block all website traffic in China?

This action must have been unintentional. 65.49.2.178 is owned by Dynamic Internet Technology according to an IP lookup, and they are behind the famous circumvention tool FreeGate. Currently, http://65.49.2.178 is a mirror site for dongtaiwang.com, a news portal operated by Falun Gong groups.

Blocking 65.49.2.178

One hypothesis is that GFW might have intended to block the IP but accidentally used that IP to poison all domains.

Hackers

Many Chinese media stated that yesterday’s outage may have been due to a hacking attempt. The IP is operated by Dynamic Internet Technology, “mortal enemy number one” of the Chinese government. Some are suggesting Dynamic Internet Technology is behind the outage. However, hacking into a root DNS resolver is not enough to cause this outage, as we explained earlier in this post. They have to hack into GFW. If they are indeed capable of doing that, they can accomplish so much more than messing the entire Chinese internet up. In addition, 65.49.2.178 during that time was not serving any content and with such traffic, it looks more like a DDOS attack agasint 65.49.2.178. They couldn't use that IP to spread sensitive content during that time. However, from today, they have indeed started to use http://65.49.2.178 to distribute mirrors and stopped within a few hours.

Blocking our mirror sites

Our mirror site for FreeWeibo has attracted considerable attention and GFW has tried multiple times to block us. We automatically rotate backend servers and the GFW automatically scans new URLs and DNS poisons them.  DNS poisoning is not commonly used compared to connection reset. GFW seems to only use DNS poisoning as a last resort when connection reset fails to block a site. Our mirror forces GFW to add hundreds of rule-sets to DNS poisoning daily and perhaps because of this we were responsible for the system crashing. This is supported by the fact that our new backend domains are no longer automatically blocked.

We’re also continuously improving our backends to prevent automatic discovery from GFW. Perhaps the script operated by GFW acquired a “null” domain from us and consequently blocked everything.

Comments

More Blog Posts

Subscribe to our mailing list
Show content from Blog | Google+ | Twitter | All. Subscribe to our blog using RSS.

Tue, Jul 05, 2016

GreatFire.org now testing VPN speed and stability in China

There is a commonly held belief in China that if you have a VPN that works then you should keep quiet about it. In terms of freedom of access to information, the problem with this approach is that access to knowledge suddenly is a secret. Today we are launching a project that we hope will destroy that model.

Our newest website, Circumvention Central (CC), aims to provide real-time information and data about circumvention solutions that work in China. Since 2011, we have been collecting data about blocked websites in China and now we will add data about the effectiveness of VPNs and other circumvention tools.

We are launching CC with four main objectives in mind.

Our first objective is to help to grow the number of Chinese who circumvent censorship restrictions in China. By sharing our information and data about these tools, we hope to show a wider audience which circumvention tools are working.

Our second objective is to improve the circumvention experience for users in China by bringing transparency to tool performance. We will measure these tools on speed (how quickly popular websites are loaded) and on stability (the extent to which popular websites load successfully).

Sat, May 07, 2016

The New York Times vs. The Chinese Authorities

Could the New York Times be setting the best path forward for news organizations in China?

Thu, Feb 18, 2016

From the desk of Lu Wei: Apple, encryption and China

Lu Wei, Director of the Cyberspace Administration of China, offers some friendly help to FBI Director James Comey.

Thu, Sep 24, 2015

Apple blocked CNNIC CA months after MITM attacks

In March of this year, Google found unauthorized digital certificates for several Google domains. The root certificate authority for these domains was the China Internet Network Information Center (CNNIC). CNNIC was controlled by the Chinese government through the Ministry of Industry and Information Technology and is now under the management of the Cyberspace Administration of China (CAC). CNNIC was recognized by all major browsers as a trusted Certificate Authority. If CNNIC signs a fake certificate used in a man-in-the-middle attack, no browser will warn of any unusual activity unless the certificate is pinned.

After Google found these unauthorized certificates, both Google and Firefox revoked its trust in CNNIC a few days later, a development we at GreatFire.org have adovacting for since 2013. Apple and Microsoft on the other hand, did not revoke their trust in CNNIC, nor did they make any announcements regarding the security compromise.

Wed, Sep 23, 2015

Malicious Xcode could spread via download manager Xunlei

What’s at stake?

We reported last week that popular Chinese iOS apps were compromised in an unprecedented malware attack. We discovered that the source of the infection was compromised copies of Xcode hosted on Baidu Pan. Apple has published an article urging developers to download Xcode directly from the Mac App Store, or from the Apple Developer website and validate signatures. We’ve now discovered that even if a developer uses a download link seemingly from Apple, he might still be possible to obtain a compromised copy of Xcode.

Please note that we do not have evidence that such attacks has happened. But it is an easy attack that anyone can implement.

How does it work?

This compromise happened because of Xunlei. Xunlei is the most popular download manager in China. Much of its popularity is due to the fact they can accelerate download speeds by pulling resources from other Xunlei users as well as cached copies on the Xunlei server. All of this, however, is invisible to users. Users can simply enter a regular http download address into Xunlei  download manager and the download will start. Chinese developers were using direct download addresses such as http://adcdownload.apple.com/Developer_Tools/Xcode_7/Xcode_7.dmg to download Xcode.

Subscribe to our blog using RSS.

Comments

и сюда запостил.

inspired a lot from this post am following this blog regularly and found very good for bookmarking thanks admin
new year sms in hindi 2015
happy new year sms 2015
happy new year 2015 wallpapers
happy new year 2015 quotes
happy new year 2015
happy new year wishes 2015

this post is awesome, great msg for us, plz update ur blog for daily basis, i am regular visitor of this site, so keep posting for us,

click the below links to create backlink
best free backlink website
click here for msg movie

thanks for this post, keep it up for updating us, i am waiting for ur new article.
IPL 2015 Cricket live score
mpscomputers
Harjinder Singh
thanks again

Add new comment

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
By submitting this form, you accept the Mollom privacy policy.