Internet outage in China on Jan 21

Yesterday we witnessed one of the largest Internet outages ever in China. We have three theories about why this outage may have occurred - two related to the Falun Gong but our third theory is that the Chinese authorities set out to attack our unblockable mirror websites.

From 15:30 to 16:30 (China time) on January 21, DNS lookup to any domain would incorrectly resolve to 65.49.2.178. Websites inside and outside of China were affected. Even Baidu and Sina were inaccessible. Only software using IP directly (e.g. QQ, VPNs) worked during that time. Attempts to visit any website redirected to http://65.49.2.178, which didn’t respond during that time.  The overwhelming traffic to this IP likely crashed the server.

Timeline

Event

15:15

GFW DNS poisoning begins. First recorded instance.

15:17

Local DNS servers began to cache incorrect responses. Some large websites in China began to be affected e.g Sina Weibo.

 

Incorrect DNS continue to spread through Chinese DNS servers. Major websites including Baidu, Sina affected.

15:39

DNS poisoning lifted by GFW. But local DNS resolvers cached incorrect responses. Users continued to experience outage.

16:00

ISPs around China were manually flushing DNS caches and connections were gradually restored.

We have conclusive evidence that this outage was caused by the Great Firewall (GFW). DNS poisoning is used extensively by the GFW. Some articles that have appeared about this outage suspected that the root DNS server in China was hacked and all domains hijacked to 65.49.2.178. This could explain why DNS servers in China were poisoned. However, during that time, we see that a lookup to 8.8.8.8, a public DNS operated by Google, returned bogus results if the lookup was done from China. In fact, the Google public DNS was not poisoned; the bogus response 65.49.2.178 could only have been returned by GFW.  If the Chinese root DNS server was hacked, a DNS lookup in China via 8.8.8.8 should have returned a correct response. See the below image from Zhihu.

Our testing system is designed to detect these bogus responses by querying non-existent DNS servers outside of China. Any valid response must come via GFW. We indeed observed such behavior during that time on all domains.  

But why did GFW poison all domains and effectively block all website traffic in China?

This action must have been unintentional. 65.49.2.178 is owned by Dynamic Internet Technology according to an IP lookup, and they are behind the famous circumvention tool FreeGate. Currently, http://65.49.2.178 is a mirror site for dongtaiwang.com, a news portal operated by Falun Gong groups.

Blocking 65.49.2.178

One hypothesis is that GFW might have intended to block the IP but accidentally used that IP to poison all domains.

Hackers

Many Chinese media stated that yesterday’s outage may have been due to a hacking attempt. The IP is operated by Dynamic Internet Technology, “mortal enemy number one” of the Chinese government. Some are suggesting Dynamic Internet Technology is behind the outage. However, hacking into a root DNS resolver is not enough to cause this outage, as we explained earlier in this post. They have to hack into GFW. If they are indeed capable of doing that, they can accomplish so much more than messing the entire Chinese internet up. In addition, 65.49.2.178 during that time was not serving any content and with such traffic, it looks more like a DDOS attack agasint 65.49.2.178. They couldn't use that IP to spread sensitive content during that time. However, from today, they have indeed started to use http://65.49.2.178 to distribute mirrors and stopped within a few hours.

Blocking our mirror sites

Our mirror site for FreeWeibo has attracted considerable attention and GFW has tried multiple times to block us. We automatically rotate backend servers and the GFW automatically scans new URLs and DNS poisons them.  DNS poisoning is not commonly used compared to connection reset. GFW seems to only use DNS poisoning as a last resort when connection reset fails to block a site. Our mirror forces GFW to add hundreds of rule-sets to DNS poisoning daily and perhaps because of this we were responsible for the system crashing. This is supported by the fact that our new backend domains are no longer automatically blocked.

We’re also continuously improving our backends to prevent automatic discovery from GFW. Perhaps the script operated by GFW acquired a “null” domain from us and consequently blocked everything.

Comments

More Blog Posts

Subscribe to our mailing list
Show content from Blog | Google+ | Twitter | All. Subscribe to our blog using RSS.

Fri, Sep 04, 2015

GreatFire Q&A with Jimmy Wales on China Censorship

We have been critical of Wikipedia’s approach to censorship in the Middle Kingdom. In a recent piece for the Huffington Post, I lamented the loss of Wikipedia in China. The encyclopedia’s founder, Jimmy Wales, who is also a staunch and public anti-censorship champion, reached out to us on Twitter. Jimmy agreed to publish our unedited exchange on the difficult nature of dealing with censorship in China.

Wed, Aug 26, 2015

Chinese developers forced to delete softwares by police

What happened?

ShawdowSocks

On August 22, an open source project called ShadowSocks was removed from GitHub.

ss.png

According to the project’s author, the police contacted him and asked him to stop working on the tool and to remove all of the code from GitHub.

police.png

He later removed the reference of the police, presumably under the pressure of the police.
edited.png

After the news, many Chinese and foreign developers, as well as ShadowSocks users, paid tribute to the author. As a result of this attention, ShadowSocks became the top trending project on GitHub.

Github.png

 

Wed, Jul 15, 2015

LinkedIn: technological and financial giants; but morally pygmies

When LinkedIn decided to create a China-hosted version of its website in February, 2014, it made a decision to compromise the company's values in the pursuit of the dollar.

It's important to note that before LinkedIn launched LingYing (the local version of the site), LinkedIn was already active in China. By their own account, they had four million registered users (with little marketing effort), a Chinese-language interface and China-based clients who were buying recruitment ads on the platform (the major source of their revenue). The site had been blocked by the authorities for one 24-hour period but otherwise was always accessible.

So why was it necessary for LinkedIn to create a local entity in China? With a local entity the company would be able to issue official receipts in RMB, making it more convenient for local companies to buy advertising on the site. A local entity also makes it easier to secure marketing deals to promote LingYing in China.

But perhaps the biggest appeal in creating a local entity for LinkedIn is that it would be among the few foreign internet companies who could cosy up with Lu Wei and the Cyberspace Administration of China (CAC). Having that kind of a relationship with CAC surely helps the business and those who are associated with the company.

Thu, Jun 18, 2015

We Had Our Arguments, But We Will Miss You Wikipedia

Wikipedia is the latest nail in the internet freedom coffin and it certainly will not be the last. Wikipedia thought that by engaging with China, the authorities would gradually open up. They thought that by allowing the Chinese authorities to censor as much information as they wanted, that eventually they would relinquish control. They thought that for those in China, having access to some Wikipedia pages was better than having access to none.

Sat, Apr 04, 2015

CNNIC censors news about their own statement

On April 1, 2015 Google announced that they will no longer recognize the CNNIC Root and EV (extensive validation) certificate authorities (CAs).

On April 2, 2015 Mozilla concluded that CNNIC’s behaviour in issuing an unconstrained intermediate certificate to another company was ‘egregious practice’ and that Mozilla products would no longer trust any certificate issued by CNNIC’s roots. Mozilla also published a more detailed report about their actions.

After unauthorized digital certificates for several Google domains were exposed by Google and Mozilla on March 23, 2015, CNNIC censored any mention of these posts. CNNIC is not only a certificate authority, they are also China’s online censorship apparatus. CNNIC was, is and will continue to practice internet censorship.

 

News about the April 1 and 2 annoucements has again been censored on social media and also on traditional media in China.

Below is a screenshot of Weibo posts about these announcements.

 

Subscribe to our blog using RSS.

Comments

и сюда запостил.

inspired a lot from this post am following this blog regularly and found very good for bookmarking thanks admin
new year sms in hindi 2015
happy new year sms 2015
happy new year 2015 wallpapers
happy new year 2015 quotes
happy new year 2015
happy new year wishes 2015

this post is awesome, great msg for us, plz update ur blog for daily basis, i am regular visitor of this site, so keep posting for us,

click the below links to create backlink
best free backlink website
click here for msg movie

thanks for this post, keep it up for updating us, i am waiting for ur new article.
IPL 2015 Cricket live score
mpscomputers
Harjinder Singh
thanks again

Add new comment

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
By submitting this form, you accept the Mollom privacy policy.