Internet outage in China on Jan 21

Yesterday we witnessed one of the largest Internet outages ever in China. We have three theories about why this outage may have occurred - two related to the Falun Gong but our third theory is that the Chinese authorities set out to attack our unblockable mirror websites.

From 15:30 to 16:30 (China time) on January 21, DNS lookup to any domain would incorrectly resolve to 65.49.2.178. Websites inside and outside of China were affected. Even Baidu and Sina were inaccessible. Only software using IP directly (e.g. QQ, VPNs) worked during that time. Attempts to visit any website redirected to http://65.49.2.178, which didn’t respond during that time.  The overwhelming traffic to this IP likely crashed the server.

Timeline

Event

15:15

GFW DNS poisoning begins. First recorded instance.

15:17

Local DNS servers began to cache incorrect responses. Some large websites in China began to be affected e.g Sina Weibo.

 

Incorrect DNS continue to spread through Chinese DNS servers. Major websites including Baidu, Sina affected.

15:39

DNS poisoning lifted by GFW. But local DNS resolvers cached incorrect responses. Users continued to experience outage.

16:00

ISPs around China were manually flushing DNS caches and connections were gradually restored.

We have conclusive evidence that this outage was caused by the Great Firewall (GFW). DNS poisoning is used extensively by the GFW. Some articles that have appeared about this outage suspected that the root DNS server in China was hacked and all domains hijacked to 65.49.2.178. This could explain why DNS servers in China were poisoned. However, during that time, we see that a lookup to 8.8.8.8, a public DNS operated by Google, returned bogus results if the lookup was done from China. In fact, the Google public DNS was not poisoned; the bogus response 65.49.2.178 could only have been returned by GFW.  If the Chinese root DNS server was hacked, a DNS lookup in China via 8.8.8.8 should have returned a correct response. See the below image from Zhihu.

Our testing system is designed to detect these bogus responses by querying non-existent DNS servers outside of China. Any valid response must come via GFW. We indeed observed such behavior during that time on all domains.  

But why did GFW poison all domains and effectively block all website traffic in China?

This action must have been unintentional. 65.49.2.178 is owned by Dynamic Internet Technology according to an IP lookup, and they are behind the famous circumvention tool FreeGate. Currently, http://65.49.2.178 is a mirror site for dongtaiwang.com, a news portal operated by Falun Gong groups.

Blocking 65.49.2.178

One hypothesis is that GFW might have intended to block the IP but accidentally used that IP to poison all domains.

Hackers

Many Chinese media stated that yesterday’s outage may have been due to a hacking attempt. The IP is operated by Dynamic Internet Technology, “mortal enemy number one” of the Chinese government. Some are suggesting Dynamic Internet Technology is behind the outage. However, hacking into a root DNS resolver is not enough to cause this outage, as we explained earlier in this post. They have to hack into GFW. If they are indeed capable of doing that, they can accomplish so much more than messing the entire Chinese internet up. In addition, 65.49.2.178 during that time was not serving any content and with such traffic, it looks more like a DDOS attack agasint 65.49.2.178. They couldn't use that IP to spread sensitive content during that time. However, from today, they have indeed started to use http://65.49.2.178 to distribute mirrors and stopped within a few hours.

Blocking our mirror sites

Our mirror site for FreeWeibo has attracted considerable attention and GFW has tried multiple times to block us. We automatically rotate backend servers and the GFW automatically scans new URLs and DNS poisons them.  DNS poisoning is not commonly used compared to connection reset. GFW seems to only use DNS poisoning as a last resort when connection reset fails to block a site. Our mirror forces GFW to add hundreds of rule-sets to DNS poisoning daily and perhaps because of this we were responsible for the system crashing. This is supported by the fact that our new backend domains are no longer automatically blocked.

We’re also continuously improving our backends to prevent automatic discovery from GFW. Perhaps the script operated by GFW acquired a “null” domain from us and consequently blocked everything.

Comments

More Blog Posts

Subscribe to our mailing list
Show content from Blog | Google+ | Twitter | All. Subscribe to our blog using RSS.

Mon, Jan 26, 2015

An Open Letter to Lu Wei and the Cyberspace Administration of China

January 26, 2015

Beijing, China

 

Mr. Lu Wei

Director of the Cyberspace Administration of the People’s Republic of China 中央网络安全和信息化领导小组办公室主任

Director of the State Internet Information Office 国家互联网信息办公室主任

Deputy Director of the Central Propaganda Department of the Chinese Communist Party 中共中央宣传部副部长

Cyberspace Administration of China,

Floor 1, Building 1,

Software Park, Chinese Academy of Sciences,

4 South 4th Street, Zhongguancun,

Beijing, China, 100190

 

Dear Mr. Lu,

On January 22, 2015, the Cyberspace Administration of China (CAC), which is under your direct control, wrote a response to a story we published about an MITM attack on Microsoft. In the post, your colleague, Jiang Jun, labelled our accusations as "groundless" and  "unsupported speculation, a pure slanderous act by overseas anti-China forces".

We at GreatFire.org take great offense to these comments and we will refute them in this letter.

Mon, Jan 19, 2015

Outlook grim - Chinese authorities attack Microsoft

On January 17, we received reports that Microsoft’s email system, Outlook (which was merged with Hotmail in 2013), was subjected to a man-in-the-middle (MITM) attack in China.

The following screenshot shows what happens when a Chinese user accesses Outlook via an email client (in this case, Ice-dove):

We have tested Outlook to verify the attack and have produced the same results. IMAP and SMTP for Outlook were under a MITM attack. Do note however that the web interfaces (https://outlook.com and https://login.live.com/ ) were not affected. The attack lasted for about a day and has now ceased.

This form of attack is especially devious because the warning messages users receive from their email clients are much less noticeable than the warning messages delivered to modern browsers (see screenshot at the end of this post for comparison).

(Sample error message from default iPhone mail client)

Fri, Jan 09, 2015

GFW upgrade fail - visitors to blocked sites redirected to porn

In the past, the Chinese authorities’ DNS poisoning system would direct Chinese internet users who were trying to access Facebook, Twitter and other blocked websites (without the use of a circumvention tool) to a set of fake IP addresses that are blocked in China or are non-existent. After waiting for some time, Chinese internet users would receive a timeout message if they were trying to access a blocked site.

However, with the new DNS poisoning system, in addition to those IP addresses used before, the Chinese authorities are using real IP addresses that actually host websites and are accessible in China. For example, https://support.dnspod.cn/Tools/tools/ shows that if a user tries to access Facebook from China, they might instead land on a random web page, e.g. http://178.62.75.99

Below is a screenshot by a Chinese user when he was trying to access our GreatFire.org website which was blocked in China. He was redirected to a goverment site in Korea. In essense, GFW is sending Chinese users to DDOS the Korea government's website.

One Chinese Internet user reported to us that when he tried to access Facebook in China, he was sent to a Russian website, unrelated to Facebook. Another user tweeted that he was redirected to an German adult site when he tried to access a website for a VPN.

某墙你这什么意思,DNS 污染返回给我一个德国工口站的 IP,满屏很黄很暴力弹弹弹(

— nil (@xierch) January 4, 2015

Wed, Dec 31, 2014

CNNIC leadership change coincides with blocking of Gmail

On December 26, 2014, in an announcement posted on their website, a new chairperson for CNNIC was directly appointed by the Cyberspace Administration of China. The announcement of this appointment coincided with the complete blocking of Gmail.

Cyberspace Administration of China (中央网信办) is chaired by Lu Wei, “China’s web doorkeeper”. Lu Wei is also the vice chair of the Central Propaganda Department, according to his official resume.

chair.png

This office is directly responsible for the blocking of Gmail and other websites including Facebook, Twitter and Google.

CNNIC is China’s certification authority and operates the country’s domain name registry. 

What are certificates used for?

Certificates are used primarily to verify the identity of a person or device, authenticate a service, or encrypt files. 

What is a certification authority (CA)?  

Tue, Dec 30, 2014

Gmail completely blocked in China

All Google products in China have been severely disrupted since June of this year and Chinese users have not been able to access Gmail via its web interface since the summer. However, email protocols such as IMAP, SMTP and POP3 had been accessible but are not anymore. These protocols are used in the default email app on iPhone, Microsoft Outlook on PC and many more email clients.

On December 26, GFW started to block large numbers of IP addresses used by Gmail. These IP addresses are used by IMAP/SMTP/POP3. Chinese users now have no way of accessing Gmail behind the GFW. Before, they could still send or receive emails via email clients even though Gmail's web interface was not accessible. 

Google's own traffic chart shows a sharp decline of Chinese traffic to Gmail. 

Below is a ping request to the Gmail SMTP server, which is completely inaccessible in China.

 

Subscribe to our blog using RSS.

Comments

и сюда запостил.

inspired a lot from this post am following this blog regularly and found very good for bookmarking thanks admin
new year sms in hindi 2015
happy new year sms 2015
happy new year 2015 wallpapers
happy new year 2015 quotes
happy new year 2015
happy new year wishes 2015

Add new comment

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
By submitting this form, you accept the Mollom privacy policy.