DDoS Attack Post-Mortem

Attack Overview

On March 13th at 7 PM CST, Wallet Guard was subjected to a large-scale DDoS attack that peaked at 600 million requests per minute over Tor exit nodes in tandem with a bot attack on our Twitter account. The attack is still ongoing.

The Wallet Guard team began implementing hotfixes, which the attacker quickly reacted to by changing their attack patterns. Ultimately, the Wallet Guard team was able to mitigate the attack entirely via AWS WAF rules that counteracted the botnet for the time being.

We believe this attack was motivated due to the recent introduction of wallet drainers that leverage active Blur approvals to drain a victim in one transaction. This attack is not just on us but all of end-user security; as soon as we mitigated the attackers, they switched over to attacking another security extension, JoinFire and began botting our Twitter account with followers to get it suspended to cut off communication to our users.

Timeline

8:13 pm, March 13th (EST)

We noticed a significant uptick in usage on our API marking the start of the attack.

10:36 am, March 14th (EST)

We implemented initial WAF rules to suspend the attack. This successfully blocked all abnormal application traffic, bringing the service back to a normal state. From 10:36 am-1:08 pm, we received 91 billion requests. The attacker sent up to 671M requests per minute during this time.

1:25 pm, March 14th (EST)

The attacker shuts down the initial attack vector.

2:18 pm, March 14th (EST)

We made a public announcement to inform users of potential service outages.

2:19 pm-5:20 pm, March 14th (EST)

The attacker restarts the process with a new technique; we were forced to take down the server to re-coordinate our defense strategy.

5:20 pm and beyond, March 14th (EST)

We re-enabled our services to witness that the attacker was still sending 100,000–200,000 requests per minute, but they are now being blocked by WAF rules as shown by a large number of 4XX response codes. The request count is now mostly stable with short spikes as we trial and error adding and removing certain rules.

9:16 pm, March 14th (EST)

Another transaction simulator extension @_joinfire goes under attack.

10:23 am, March 15th (Latest update)

The attack is still ongoing; however, our current measures have successfully mitigated with normal traffic returning to the services. In this time, we received over 29 million requests.

How did we address the attack?

To effectively combat the attack, the Wallet Guard team employed a multi-layered approach using AWS WAF rules tailored to mitigate the impact of the botnet. Our strategy included the following steps:

1. Rate Limiting: We enforced strict rate limits on incoming requests to prevent the flood of traffic from overwhelming our systems.
2. IP Blocking: We identified and blocked malicious IP addresses associated with the attack, particularly those originating from Tor exit nodes.
3. Anomaly Detection: Our team closely monitored traffic patterns and swiftly reacted to any anomalies, adapting our defense measures as necessary.
4. Advanced Filtering: We implemented custom rules to filter out illegitimate requests.
5. AWS Managed Rules: We enabled AWS WAF managed rule sets designed to protect against common attack vectors, such as SQL injection and cross-site scripting.
6. Infrastructure Scaling: We dynamically scaled our infrastructure horizontally to handle the increased traffic load and maintain service availability for legitimate users.

Lessons learned?

The massive DDoS attack on Wallet Guard provided valuable insights and lessons that will help improve our security posture moving forward. Some of the key takeaways include:

1. Expect the unexpected: Even with robust security measures in place, it’s crucial to remain vigilant and prepared for potential attacks. The scale and adaptability of the attack demonstrated the importance of having a flexible and responsive incident management plan.
2. Continuous monitoring and analysis: Regularly monitoring and analyzing traffic patterns, logs, and infrastructure health is essential for the early detection of attacks and swift response to evolving threats.
3. Layered defense strategy: Implementing a multi-layered defense approach, combining services like AWS WAF, and AWS Shield proved effective in mitigating the attack. This approach helps ensure that even if one layer is compromised, others can still provide protection.
4. Incident response planning: Having a well-defined incident response plan is critical for effectively managing a security crisis. The plan should include roles and responsibilities, communication protocols, and guidelines for decision-making during an attack.
5. Collaboration and communication: With this as our first major tech incident at Wallet Guard, we learned how to communicate across multiple time zones while delegating incident response work items. We’ve learned how important it is to stay in touch with your community during an attack. We will continue leveraging Twitter & Discord to relay updates live.

By reflecting on these lessons and applying them to our security strategy, Wallet Guard is better equipped to prevent, detect, and respond to future threats, ensuring the safety and reliability of our services for our users.

Technical Overview

Wallet Guard was subjected to a highly sophisticated and large-scale DDoS attack on March 13th, which involved a peak rate of 600 million requests per minute and over 200k unique IP addresses, predominantly originating from Tor exit nodes. The attack targeted both the application layer (Layer 7) and the network layer (Layer 3), employing multiple attack vectors and demonstrating a high level of adaptability.

1. Multi-Layered Attack Type: The DDoS attack targeted both the application layer (Layer 7) and the network layer (Layer 3). At the application layer, the attacker launched a volumetric attack at the network layer to saturate network bandwidth and disrupt infrastructure.
2. Attack Vectors: The attacker employed multiple attack vectors, including HTTP floods to overwhelm Wallet Guard’s infrastructure. This multi-vector approach made it more challenging to detect and mitigate the attack effectively.
3. Tor Exit Nodes: The attacker leveraged Tor exit nodes to obfuscate their identity and bypass conventional IP-based blocking techniques. This strategy increased the difficulty of tracing the attacker’s true origin and required more advanced mitigation techniques.
4. Botnet: The scale of the attack suggests that the attacker-controlled a large botnet composed of numerous compromised devices. The botnet allowed the attacker to generate massive amounts of traffic and sustain the attack for an extended period.
5. Attack Duration: The attack persisted for several hours, with fluctuations in intensity, before the Wallet Guard team successfully mitigated it using a combination of AWS WAF rules and other defensive measures.

No user data was exposed or impacted by this attack. The impact was strictly service interruptions on our transaction simulation feature within the Wallet Guard browser extension.

Closing thoughts

Throughout the ordeal, our team has gained valuable insights that better equip us for the next attack. This attack only strengthened our belief that end-user security is required. We are as committed as ever to delivering solutions that help us all stay one step ahead of an attacker. By leveraging AWS WAF’s powerful features and adapting to the evolving threat landscape, we were able to successfully mitigate the attack and protect the integrity of Wallet Guard’s services.

Website: https://walletguard.app
Twitter: https://twitter.com/wallet_guard