Internet outage in China on Jan 21

Yesterday we witnessed one of the largest Internet outages ever in China. We have three theories about why this outage may have occurred - two related to the Falun Gong but our third theory is that the Chinese authorities set out to attack our unblockable mirror websites.

From 15:30 to 16:30 (China time) on January 21, DNS lookup to any domain would incorrectly resolve to 65.49.2.178. Websites inside and outside of China were affected. Even Baidu and Sina were inaccessible. Only software using IP directly (e.g. QQ, VPNs) worked during that time. Attempts to visit any website redirected to http://65.49.2.178, which didn’t respond during that time.  The overwhelming traffic to this IP likely crashed the server.

Timeline

Event

15:15

GFW DNS poisoning begins. First recorded instance.

15:17

Local DNS servers began to cache incorrect responses. Some large websites in China began to be affected e.g Sina Weibo.

 

Incorrect DNS continue to spread through Chinese DNS servers. Major websites including Baidu, Sina affected.

15:39

DNS poisoning lifted by GFW. But local DNS resolvers cached incorrect responses. Users continued to experience outage.

16:00

ISPs around China were manually flushing DNS caches and connections were gradually restored.

We have conclusive evidence that this outage was caused by the Great Firewall (GFW). DNS poisoning is used extensively by the GFW. Some articles that have appeared about this outage suspected that the root DNS server in China was hacked and all domains hijacked to 65.49.2.178. This could explain why DNS servers in China were poisoned. However, during that time, we see that a lookup to 8.8.8.8, a public DNS operated by Google, returned bogus results if the lookup was done from China. In fact, the Google public DNS was not poisoned; the bogus response 65.49.2.178 could only have been returned by GFW.  If the Chinese root DNS server was hacked, a DNS lookup in China via 8.8.8.8 should have returned a correct response. See the below image from Zhihu.

Our testing system is designed to detect these bogus responses by querying non-existent DNS servers outside of China. Any valid response must come via GFW. We indeed observed such behavior during that time on all domains.  

But why did GFW poison all domains and effectively block all website traffic in China?

This action must have been unintentional. 65.49.2.178 is owned by Dynamic Internet Technology according to an IP lookup, and they are behind the famous circumvention tool FreeGate. Currently, http://65.49.2.178 is a mirror site for dongtaiwang.com, a news portal operated by Falun Gong groups.

Blocking 65.49.2.178

One hypothesis is that GFW might have intended to block the IP but accidentally used that IP to poison all domains.

Hackers

Many Chinese media stated that yesterday’s outage may have been due to a hacking attempt. The IP is operated by Dynamic Internet Technology, “mortal enemy number one” of the Chinese government. Some are suggesting Dynamic Internet Technology is behind the outage. However, hacking into a root DNS resolver is not enough to cause this outage, as we explained earlier in this post. They have to hack into GFW. If they are indeed capable of doing that, they can accomplish so much more than messing the entire Chinese internet up. In addition, 65.49.2.178 during that time was not serving any content and with such traffic, it looks more like a DDOS attack agasint 65.49.2.178. They couldn't use that IP to spread sensitive content during that time. However, from today, they have indeed started to use http://65.49.2.178 to distribute mirrors and stopped within a few hours.

Blocking our mirror sites

Our mirror site for FreeWeibo has attracted considerable attention and GFW has tried multiple times to block us. We automatically rotate backend servers and the GFW automatically scans new URLs and DNS poisons them.  DNS poisoning is not commonly used compared to connection reset. GFW seems to only use DNS poisoning as a last resort when connection reset fails to block a site. Our mirror forces GFW to add hundreds of rule-sets to DNS poisoning daily and perhaps because of this we were responsible for the system crashing. This is supported by the fact that our new backend domains are no longer automatically blocked.

We’re also continuously improving our backends to prevent automatic discovery from GFW. Perhaps the script operated by GFW acquired a “null” domain from us and consequently blocked everything.

Comments

More Blog Posts

Subscribe to our mailing list
Show content from Blog | Google+ | Twitter | All. Subscribe to our blog using RSS.

Fri, Mar 27, 2015

CNNIC censored Google and Mozilla’s posts about CNNIC CA

This week, Google found unauthorized digital certificates for several Google domains, the root CA of which is CNNIC. Google and Mozilla both publicly disclosed this security incident and published blog posts(Google, Mozilla). However, Chinese translations of Google’s and Mozilla's blog posts were censored on the Chinese Internet.

  • William Long is a prominent Chinese blogger on IT and tech. He translated Google’s security post without adding any personal opinions. The Chinese blogpost ranked #1 when searching CNNIC MITM in Chinese on Google and Baidu. He tweeted that he received a phone call from propaganda department demanding the post to be removed immediately. The post http://www.williamlong.info/archives/4183.html was deleted. Google cache is still available.

Wed, Mar 25, 2015

Evidence shows CNNIC and CAC behind MITM attacks

Since 2013, we have repeatedly called on major software vendors to revoke CNNIC-issued certificates. Most notably, we raised this issue when we reported on the Cyberspace Administration of China’s (CAC) man-in-the-middle (MITM) attacks on Google, Microsoft’s Outlook, Apple, Yahoo and Github. Mainstream media have reported about these security vulnerabilities before and on March 24, Ars Technica reported on Google’s announcement that they have definitive evidence that CNNIC (China Internet Network Information Center) was behind a new MITM attack on Google.

From our October, 2014 blog post:

Thu, Mar 19, 2015

We are under attack

We are under attack and we need help.

Likely in response to a recent story in the Wall Street Journal (WSJ), we’ve experienced our first ever distributed denial of service (DDoS) attack. This tactic is used to bring down web pages by flooding them with lots of requests - at the time of writing they number 2.6 billion requests per hour. Websites are not equipped to handle that kind of volume so they usually “break” and go offline.

This kind of attack is aggressive and is an exhibition of censorship by brute force. Attackers resort to tactics like this when they are left with no other options.

We are not equipped to handle a DDoS attack of this magnitude and we need help. Some background:

  • The attack started on March 17 and we are receiving up to 2.6 billion requests per hour which is about 2500 times more than normal levels.

Thu, Mar 12, 2015

Collateral Freedom and the not-so-Great Firewall

Recognizing that the authorities have been hesitant to crackdown on our method of circumvention, we have accelerated our expansion of the development of collateral freedom, in three key areas.

Mon, Jan 26, 2015

An Open Letter to Lu Wei and the Cyberspace Administration of China

January 26, 2015

Beijing, China

 

Mr. Lu Wei

Director of the Cyberspace Administration of the People’s Republic of China 中央网络安全和信息化领导小组办公室主任

Director of the State Internet Information Office 国家互联网信息办公室主任

Deputy Director of the Central Propaganda Department of the Chinese Communist Party 中共中央宣传部副部长

Cyberspace Administration of China,

Floor 1, Building 1,

Software Park, Chinese Academy of Sciences,

4 South 4th Street, Zhongguancun,

Beijing, China, 100190

 

Dear Mr. Lu,

On January 22, 2015, the Cyberspace Administration of China (CAC), which is under your direct control, wrote a response to a story we published about an MITM attack on Microsoft. In the post, your colleague, Jiang Jun, labelled our accusations as "groundless" and  "unsupported speculation, a pure slanderous act by overseas anti-China forces".

We at GreatFire.org take great offense to these comments and we will refute them in this letter.

Subscribe to our blog using RSS.

Comments

и сюда запостил.

inspired a lot from this post am following this blog regularly and found very good for bookmarking thanks admin
new year sms in hindi 2015
happy new year sms 2015
happy new year 2015 wallpapers
happy new year 2015 quotes
happy new year 2015
happy new year wishes 2015

this post is awesome, great msg for us, plz update ur blog for daily basis, i am regular visitor of this site, so keep posting for us,

click the below links to create backlink
best free backlink website
click here for msg movie

thanks for this post, keep it up for updating us, i am waiting for ur new article.
IPL 2015 Cricket live score
mpscomputers
Harjinder Singh
thanks again

Add new comment

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
By submitting this form, you accept the Mollom privacy policy.