Two Leaks in the Great Firewall: Hacking DNS Injectors

[00:00:15]Hi everyone. It's an honor to be here to uh talk to Congress. [clears throat] Uh I'm here to tell you about two leaks in the Great Firewall um and how they tie together and you know what we can do uh to sort of further this sort of research.

[00:00:31]So just to give an introduction to myself, I'm Jade. I'm a PhD student at the University of Massachusetts Amherst at Spin. Uh my adviser is Amir Monsider.

[00:00:39]Uh you should definitely go apply to work with him if you're interested in this stuff. Um so I work mostly in internet censorship measurement and internet censorship circumvention.

[00:00:49]So just to give a background of, you know, how the great firewall works, what it is. So most people probably know that in China you can't go to a lot of websites. They're blocked uh via different mechanisms. HTTP, TLS, DNS.

[00:01:01]Um, but in general, they don't like you going to websites that they don't like.

[00:01:05]So, you know, the usual thing is something like an HTTP or TLS reset, where you try to access a website and it will send you a bunch of TCP resets saying, "No, this connection is terminated. You're not allowed to go here." Uh, but in this talk, we're going to focus on their DNS injector. So, the way the DNS injector works is you send a DNS request for a website, say Google.com. um that DNS request will actually go to the real resolver. Um and the injectors will send back fake responses with bogus IP addresses, totally not real. Um and you won't be able to go to the website cuz TLS will break. Um and the IPs don't even resolve in China. They're actually black holed at a national level mostly.

[00:01:47]Um yeah, so the interesting thing is there's not just one injector in China, but actually three. So when you send a request, you can actually get up to three injections.

[00:01:57]Um and all of these injectors have different characteristics. Uh in this talk we'll mainly focus on injector 3.

[00:02:05]So even with the cases of the multiple injectors your DNS packet actually isn't dropped at the final destination. Uh the real resolver does get that packet.

[00:02:15]So the way the bogus IPs work is when you uh send the query you get a IP. It's totally blocked. Um but something interesting about these IPs is actually some of them do resolve. Uh we noticed uh a little bit over the last couple years stopping earlier this year that these IPs would respond to a TCP handshake and nothing else. Um and some of them actually uh hosts an adult website. So some percentage of requests for uh blocked websites in China will actually lead you to porn.

[00:02:49]Uh, another interesting thing about the injector is that it's birectional.

[00:02:53]So if you, you know, send a request and you get this injected response from China, you'd expect that. Of course, you can actually do this from America or from Germany probably. I didn't test it on this Wi-Fi, but you can try it. Uh, if you send an NS lookup for something that's blocked in China to most Chinese IPs, uh, you'll actually get an injected response. So you can play with this injector anywhere. Um it makes it really nice for experimentation.

[00:03:21]So yeah, these are the characteristics of the injectors. Um as I said, we'll be focusing on injector 3. Um and these are just some fingerprints. So you can tell like there's different uh IP and DNS level flags on these packets and that's how we tell which injector is which.

[00:03:38]So onwards to the actual subject of the talk or the main subject, wall bleed.

[00:03:44]So while beta is an academic work, I want to give a shout out to the actual authors. Uh I I made a contribution to this work but definitely did not lead it. Um so that's Jen Shaan of GFW report. Uh Jackson Sip of UC Boulder, Sakamotoan of Shinon Nome Lab, David Ffield, Amir Hman Solder from UMass Amherst, um Elsen Wedwards and Eric Wro of UC Boulder. If you're interested in just reading this paper, it is here.

[00:04:12]So just give an intro of you know DNS.

[00:04:15]The packet structure for DNS is very simple. So you start with a transaction ID that says you know what is the ID of my request and you when you get one back of that same ID you assume that that's what you asked for. So this is how the injectors work. Um you have some flags the number of questions. We'll skip those since the answers we don't need necessarily right now. But the name length is the interesting part. So DNS names are segmented into lengths and strings. So you have for example 8 Facebook 3 com and then a terminator zero. Um and then you have you know the type internet or type a and class in for internet.

[00:04:55]So what is wly? Uh so regularly here's what the traffic looks like uh when you send an injected response.

[00:05:05]So, you'll send a request for a website and you'll get back one of these bogus IPs. That IP is Dropbox, by the way.

[00:05:12]Clearly not Facebook. Um, and that's blocked anyway. So, it's like why wobbly.

[00:05:20]So, if you change the length, you notice we had 8 Facebook 3 com, but now we have 8 Facebook0x20 com. It started dumping memory.

[00:05:34]And that memory is UPN traffic which shouldn't be on the public internet.

[00:05:39][laughter] So we started with this basically and we did some refinements to see let's get the maximum amount of memory we can out of these things.

[00:05:54]So first uh we shortened our domain. So we found that 3.tt TT would actually trigger the injector and we could get vulnerable responses with this very short domain and so that gives us more space for activities.

[00:06:10]Uh it also turns out that you could just remove the Q type and Q class at the end and it it's fine.

[00:06:18]And of course you know we could actually take that length and change it to 0x FF and as a result we actually get up to 124 bytes every time we send a wobbling query.

[00:06:30]So there were some uh setbacks in the process. So 3.tt uh was the original domain used to trigger the injector. Um and they eventually stopped blocking it for whatever reason. Maybe it fell off the block list. Maybe the website died.

[00:06:44]Uh so we just changed it to 4.TT and that worked. Uh so we also had an event where in September 2023 somewhere in that range uh they actually patched Wbleleed. So they had uh all of the IPs that were responding for WB just completely stopped within a couple weeks or so. Um and we assume this is how long it took them to patch it.

[00:07:11]Uh so we just made some modifications and we put the QT uh QT type and QC class back and it worked and we moved the label a little bit.

[00:07:22]Um that got patched within a couple weeks as well but we gave us more time to do experiments with so to show you know what leaked what happened we did uh a longitudinal experiment where we sent probes from UMass to a 10-centent cloudVPN. So it's our VPN we control it.

[00:07:39]We're not dosing some random person. Um, and we sent it over UDP with varied ports to port 53 and we sent 100 packets per second for about two years and got 5.1 billion WLED responses.

[00:07:56]So when we looked at this, we're like this is real traffic [laughter] uh from people in China like using the internet [snorts] uh including stuff that shouldn't be there, right? You'd think that a DNS injector only handles DNS packets. Nope. Uh it had all kinds of stuff like uh UPN for example. Um and you might think, oh, you know, this is crazy unethical. Are you spying on the Chinese citizens?

[00:08:23]Well, no. We we're not that kind of people. Um we did some initial manual inspections to see what we were working with. But all further analysis we did with regular expressions, you know, we wanted to keep things ethical.

[00:08:35]Uh but what's one of the interesting things that was uh we noticed is that this could actually be used for some sort of roughly targeted surveillance.

[00:08:45]So I mentioned earlier that if you send to basically any IP in China, you can trigger these injectors. It turns out if you pick a specific IP, you can get injectors near that location.

[00:08:55]So I'm sure some nation state was using this long before we found it. Uh but that's speculation, not only my opinion.

[00:09:03]Um, so one of the things we notice frequently in this traffic is SSDP, which shouldn't be on the public internet. Um, but it appeared a lot, which is weird. Uh, we also noticed a bunch of UD UPN, which is also weird that shouldn't be on the public internet.

[00:09:22]Um, and so when we looked at this, uh, we looked at the headers for the UPN packets, which had a location in it. So it was something like, you know, HTTP.

[00:09:32]um all of them were private IP addresses. So it's like what is this traffic even?

[00:09:38]Uh we also looked for packet headers. So uh in IPv4 a lot of the packets will start with uh 0x45 0x00 0. And so we were able to find like full packet stuff not just like application layer traffic.

[00:09:53]Um, so when we drilled down to see like what the IP protocol field in these packets that had the 0x450, uh, we found a ton of TCP, a ton of UDP and some other stuff that's not super common.

[00:10:07]Um, filtering down to just the TCP packets, looking at the source and destination addresses, we found that a lot of it is still private traffic and only a somewhat small portion is like public IP to public IP traffic.

[00:10:22]So maybe there's some filters in place or maybe there's just it's it's unclear. We we suspect a lot of internal management traffic for the great firewall went through the same buffer.

[00:10:34]So yeah, so a bunch of internal management traffic, right? Well, if you remember walled, you've probably seen a lot of this [laughter] or not wall bleed, uh heartbleleed.

[00:10:45]Uh we were able to find uh various, you know, what people were browsing. We were able to find cookies, um, mail, as well as people's passwords, for example. Um, lots of very unfortunate privacy leaks happening here.

[00:11:02]So, something weird we noticed early on when doing these experiments is the first four bytes of the leaked payloads would have like these seemingly random bytes, but the bytes seem to depend on the IP and uh, ports of the traffic that was sent. So we suspect it might have been some sort of like identifier um used for like tagging the traffic and load balancing it.

[00:11:25]Uh but we noticed they changed twice. So in uh September 2022 the percentage of traffic that had this uh dropped significantly and then in June 2023 it dropped even more.

[00:11:42]So we also noticed in this traffic uh some x8664 stack pointers. So, we're thinking, oh, you know, maybe this is a buffer overflow into some code. We can figure out, you know, the source code of this injector. Um, we didn't get that far. Um, it didn't seem to have that, but we have some weird patterns. So, if in cases where we found these stack pointers, we say S is a stack pointer, C is a code pointer just based on like usually what addresses these are in Linux.

[00:12:11]Uh, 1248 are some common we found frequently. uh you can check the paper for what exactly those were. Um but we just use them as codes here and then we have a wild card. So if we graph the patterns of what uh of these patterns appeared frequently we can notice two change points here and here that actually match what happened of the digestes. So based on this we are actually to figure out able to figure out when they were patching the great firewall and use this as a sort of side channel to see what was being changed.

[00:12:44]So we also, you know, just to make sure we actually weren't just seeing internal traffic, we actually did a real experiment on this. So we sent probe traffic. So we use the string GFW bleed, which surely shouldn't exist in anyone else's traffic.

[00:12:58]Um, and uh some tags to figure out, you know, where our experimental traffic came from. And we sent it from a single source IP um to port 53 and we sent it at about 30 packets per second. uh we also sent wobbble bleed packets at the same time.

[00:13:15]So when we send these queries uh sending it both at the same time and we expect to see wall bleed responses containing our own tagged traffic um we did actually at a somewhat low rate because this machine was handling the traffic of basically most of China.

[00:13:34]Uh but we noticed that the percentage we saw actually correlated with when people were awake. So clearly this was representative of real internet traffic.

[00:13:45]Uh we also from a probe in UC Boulder uh sent out probes to all over the world just sending these GFW bleed uh doing the same experiment um but to all these different arbitrary IPs around the world and we found that traffic to IPs uh using I think geolite 2 database for locating um was affected by wbleleed everywhere. basically anything that routes through China at one point ended up getting leaked somehow.

[00:14:15]Um so this is not just a threat to the users of China and their privacy but also users around the world.

[00:14:23]Um so just to give an overall the timeline so we first discovered Wall in 2021. Um eventually they removed that original domain from the block list. Um there were some patches and then the first main patch we noticed where this was fixed was uh November 2023 and then the second fix for Wall Bleed V2 was 2024. So just mainly the main dates for that to remember are in around September to November 2023 and around March 2024.

[00:14:51]So now I'm going to talk about another leak. So recently, uh, Gege Networks is a Chinese cyber security company founded by Fang Bing, the so-called great, uh, father of the great firewall. Um, and 600 GB of documents, source code, binaries, all sorts of fun internal junk was leaked from this company and you can go grab that right now. Um, uh, this was released by anonymous source via Endless Hakista. Um, and there was a lot of really great initial analysis provided by the great firewall export project and that includes Amnesty International, uh, the Intersect Lab, Tour Project, Justice for Myanmar, Paper Trail Media, Dirt Standard, Follow the Money, and the Glob and Mail. So definitely go check out that reporting.

[00:15:35]It's really good. Um, and so we've been doing some further work because obviously this is a gold mine for academic censorship researchers.

[00:15:44]So we identified um, as one of the components. So the Misa platform's DNS uh parser. So we strongly suspect that the Misa platform was what they use for the DPI within G networks. Um and we specifically looked at their DNS parser.

[00:16:02]So the idea was did these guys do wall bleed? You know, is it their fault? Can we point fingers at them and make fun of them? Um and so looking at the timeline, here's some get comments.

[00:16:14]So based on these, we might say that, you know, that looks pretty promising.

[00:16:19]Um, and then the patch after that, you know, maybe it was late, maybe they hot fixed it. Um, something like that. So these would be the comments to look for if that were what was in the um if that were the source of the leak.

[00:16:35]So probably not. Um, we looked at the wording for memory leak. I use Google Translate for this because I don't know Chinese. Um, but we think memory leak here implies like an actual memory leak like you know not freeing stuff.

[00:16:51]Um, yeah. So what the patch appears to do here is it will take the parsing logic and move it earlier in the function and so that avoids allocating a session object for the uh DNS session. Um, something funny I noticed in the source code is they call their Malik dictator Malik. Like that's a little on the nose guys, don't you think?

[00:17:16]Um but so you know looking at the part comments after that they didn't have any relevance. They were just Linux management stuff.

[00:17:25]Uh so yeah um in general censorship is getting worse across the world. And what we want to do here is note that you know these sensors can be attacked. you don't just have to, you know, measure, figure out what they're blocking, figure out how they're blocking it. You can actually just go after them. Um, and this hurts not just the sensor, but also um the users. It's harm obviously like people being in a censored country and having some like idiots running this like unsecure software just destroys your privacy as a country. Um, if you're interested in any of this stuff, definitely come talk to me. Um, and if you're already doing this stuff, come talk to me. I love getting connections.

[00:18:07]So yeah, uh if you have any questions, uh you can ask me now or uh you can email the primary author on this paper who's going to have more knowledge of this topic.

[00:18:19]Um special thanks to obviously all the authors of the original work, a special GFW report. Definitely go check their stuff out. Um and then additionally all the uh analysis of the G league. Thank you very much. [applause] Thank you.

[00:18:40]All right. No, that's more time for questions than may think. Thank you, Jade, for your talk. So, if you have questions, we have microphones over here. You can see them in the room.

[00:18:49]Please queue up if you have questions already. Otherwise, there's also the possibility to ask questions on the internet. On the IRC, it's heckend.org.

[00:18:58]the room is #9 39C3 whole uh zero or on matrix it's just hall zero I see no questions at the moment so Jade my first question you mentioned this dictator log so did you have a look into it what was the special thing was it aligned in any way or what's so it appeared to be designed specifically for uh single applications running on a server so like these are you know dedicated appliances um it seemed to be doing some like low-level allocation at the kernel level. Um, but it it didn't seem particularly special.

[00:19:33]>> Um, it's an interesting target for exploitation though and I can share anyone the path in the leak if anyone's interested in analyzing it.

[00:19:40]>> So, if some people want to talk up to you after the talk, where can they meet you?

[00:19:45]>> Um, that's I actually don't know where I'll be during congress. I probably put up my email.

[00:19:50]>> Okay, then hit up my email. Um, I see one question over there at microphone number four. Please shoot. uh thanks for your amazing work. Um it's really funny that we can talk about this topic here in public like exploiting some uh firewall service and so on. That's that's very unusual uh I think. So is this like the only opportunity to hack a censorship firewall to not get driven by court or something like this? And especially how do you handle uh that you identified that the vulnerability was maybe also abused by other actors which maybe not have that uh good intention in mind.

[00:20:31]>> Absolutely. Yeah. [laughter] Thanks. Okay.

[00:20:36]Well, thank you for the comment. Any other questions at this point? Signal Angel.

[00:20:43]>> I don't see any. So, let's thank the speaker again.

[00:20:47]>> Oh, there's one. Oh, sorry. There's one just right. So, making uh Thank you for your excellent talk. Would you ever consider traveling to China?

[00:20:58]>> Uh, I'm pretty sure, especially now that they know my face, I'm like giga banned from China. Okay. So, uh, now they are all arriving. Microphone number one.

[00:21:08]>> Hi. Sorry, I'm Mona. Um, uh, I was wondering, it's been a few months now since the gagege leaks. I'm wondering if, uh, there have been any updates or where we can learn about what folks have been finding in them or, yeah, has there been anything published, uh, I guess on this front? So, as you probably know, a lot of us are scrambling to publish stuff about this. [laughter] Uh, we're currently have a bunch of stuff that's currently under publication that I can't exactly talk about right now, but one of the interesting things we're looking into related to this is actually can we exploit GE? All of their source code is there. We know how their things work. We can find vulnerabilities.

[00:21:47]It's in there. You just have to find it.

[00:21:49]Um, and we've been looking actively into this. Um can you describe where is GI uh deployed or how is the code?

[00:21:58]>> So it's deployed actually in uh regional areas in China.

[00:22:03]>> So the way um we we we believe it's not actually the great firewall China. It's just sold to local customers. So Kongjo um there's an island off the coast of Taiwan. I forget the name of uh Myanm.

[00:22:17]But it's also deployed internationally.

[00:22:19]So J actually uh exports some of their censorship software to these other countries um sort of following along the lines of wherever the belt and road countries are and this includes Kazakhstan, Pakistan, Myanmar, um Ethiopia and there's a mystery customer that's like A21 in their code names uh that we are not sure what it is yet, but we're actively trying to figure out who's buying from this company.

[00:22:42]>> Okay, thank you so much.

[00:22:43]>> Thank you. [laughter] I see another question at microphone number six. Uh yeah, thank you again for the amazing talk. Um has there been any retaliation against China researchers either from uh China itself or domestically?

[00:22:58]>> Uh I'd rather not get into the details for the safety of the participants but unfortunately yes.

[00:23:03]>> Thank you. Okay, microphone number two.

[00:23:06]>> Thank you for your work. Um I was wondering can you say something about the level of centralization of the Chinese internet? So um in a sense that you can only see specific phenomena in one ASN or uh you have only one as in China in general. Yeah. So it's actually very diverse. Um certain areas like with the G thing actually are deployed only regionally. Um like G software seems to be deployed only to certain customers that want it. Um, in the case of Wbleleed, we believe it to be the national injector because many different IPs stopped responding. Like I think we had the figure of it being in the millions.

[00:23:48]Um, and so that's uh indicates that this was sort of a national level coordinated response. Um, so it it sort of varies.

[00:23:58]Uh, as I said, there's also the three injectors, all of which have different behaviors and different block lists.

[00:24:05]Okay.

[00:24:06]Um, yeah. Thank you for the talk. Um, I have one question about the passwords you found. You mentioned that you found these in HTTP headers or query parameters. So, does that mean that much of the traffic in China is unencrypted HTTP or did they break TLS too? Uh, I don't believe they broke TLS. Um, but I I think it's just a lot of HTTP traffic.

[00:24:33]uh they do have specific specialized resolvers and injectors for other protocols like TLS and quick and even uh TLS ESNI >> number five.

[00:24:45]>> Yeah, thank you for your talk. Um I was wondering if you had done any analysis on what sort of DNS queries would trigger the blocking and if there was anything surprising in that.

[00:24:58]>> Um I've looked at some of the categories. A lot of it ends up being like gambling or pornography or you know politically sensitive websites. Uh I can show you places to get some of these big lists of flock domains if you want afterwards. Um it's it's sort of what you would expect. There's nothing particularly uh exciting um about it in my opinion.

[00:25:23]>> Number four.

[00:25:24]>> Um thank you for great talk. Uh the thing that you mentioned there are three layers of DNS injectors really fascinated me. Uh why there are three layers maybe why not four and which each one what does the purpose each layer has. Do you know or can you share more about it? Yeah I mean it's not clear what the purpose of these injectors are and I I think there actually might be some work looking into this. Um I suspect it's just so China has you know maybe a national level firewall that says like this is deployed at all the gateways. Uh you you know these have to happen. Um but a lot of censorship is actually the Chinese government saying hey you ISP uh you need to implement censorship right now or bad things are going to happen. So you know that ISP implements censorship and then maybe another ISP implement censorship. And so if your traffic routes through any of these ISPs it will get hit by all of the injectors along that path.

[00:26:19]Thank you. I see the internet has more questions for us.

[00:26:24]>> Yes, the internet wants to know if you can tell us more about the general setup that used to analyze the traffic.

[00:26:33]>> Sorry, I missed the question.

[00:26:34]>> Can you come again?

[00:26:43]>> Okay, let's send over for to microphone number two for a second. Uh I can also repeat the question from the internet.

[00:26:50]>> Okay. Sorry. Yeah.

[00:26:52]>> Uh but that's not my question right. So it was if you can talk about the general setup of I forgot the last word.

[00:27:01]>> Uh >> uh Ge >> Yes I think >> Ge. Yeah absolutely. So something we noticed um and I think I might have forgot to put the slides in it but so they have uh their main DPI platform known as SAP. Uh and then within SAP they have a bunch of DNS parsers that appear to have formerly been known as Stellar but then they integrated it and it just became part of the platform um called MISA platform.

[00:27:24]So there is active work in trying to actually run people's own copies of that uh that's being done by other great researchers. Um we don't currently have a full understanding yet. Um but I'm sure there's going to be some publications pretty soon.

[00:27:40]Um and my question was why do you think there are different injection layers like there was injection like one two and three and you only did three y >> um why do you think they are split?

[00:27:54]>> Yeah. So as I said uh I think I think I answered this previous question but I think it's the ISPs are essentially being told to implement their own censorship system on top of the national level system and so so this might not this might be just for not just accessing traffic internationally but also internationally um and so there's a lot of layers of censorship um not just internet but rather like you know social media companies being told to do certain things selfcensorship as a result um but there doesn't appear to be coordin ordination as far as we can tell or at least much coordination between G networks and the actual national great firewall weirdly. Um but that that's speculation on my part. Uh we haven't fully confirmed this but that's the idea I get from looking through the leaks for weeks.

[00:28:39]>> Thank you.

[00:28:40]>> I've seen there's maybe another internet question.

[00:28:45]>> Yes. Uh you mentioned in your talks that the DNS traffic was changed from Facebook to Dropbox for example. Uh the question is is this part of the great firewall or what's the reason for that?

[00:28:59]>> Uh so it's just part of the great firewall. So the IPs as I uh mentioned early on are mostly completely black holed by the great firewall. Um one of the things I we noticed with the IPv4 injector for a requests is it will send all these random IPs from Facebook, Dropbox and some other random stuff. And we believe these IPs are probably related to websites they really want to censor like you know maybe human rights organizations. Uh but for the IPv6 resolutions they actually send uh mostly Facebook like like the IPs will have like FAC B O. Uh but something weird in the IPv6 resolver that you know maybe others might be interested in is they send to addresses and not just normal to addresses but broken to addresses. So, it'll have 2001, a bunch of zeros, and then an IPv4 address at the end, which is supposed to be encoded in Trito. Uh, I don't know why they do this. I think it's just they want something that doesn't resolve.

[00:30:00]They do make mistakes sometimes. Like I said, they actually redirected users to pornography.

[00:30:08]>> Another question from the internet.

[00:30:10]>> Yes. Next question. Have you by any chance also looked at firewalls from other countries like Iran or something else?

[00:30:19]>> Iran is actually difficult to get internet censorship vantage points in.

[00:30:23]Um a lot of work has been in China but we are actively looking into other countries. Um especially now that with the GE leak we've confirmed that uh China is exporting censorship not just locally but also abroad.

[00:30:37]>> Okay. [clears throat] So, thank you all the people who participated in this Q&A and thanks again Jade for the very interesting talk. Thanks.

[00:30:46]>> Thank you. [applause]