Be cautious of popular technologies that promise to make work easy - they often cut corners, and security is often one of them.

A few days ago, the database of NewsBlur (a web-based RSS reader) was deleted by a hacker due to a bug that existed in Docker for years.

In a blog post this week, NewsBlur founder Samuel Clay detailed his ordeal: during a migration of a MongoDB cluster to a Docker container, a hacker gained access to the NewsBlur database, deleted 250GB of raw data, and demanded a 0.03 BTC ransom.

Clay blamed the attack on a firewall error that occurred during the database migration: “When I containerized MongoDB, Docker helped me open up MongoDB to the world by inserting an allowable rule in iptables.”

This issue with Docker was reported to Docker as a “serious bug” back in 2014, but has not been resolved.

How was NewsBlur’s MongoDB server compromised?

Last week, Clay began maintenance work on NewsBlur as he prepared to migrate several of the company’s PostgreSQL, MongoDB, Redis, and Elasticsearch databases to Docker containers. In the hours leading up to the attack, he switched the application’s MongoDB cluster to the new server and shut down the original server, intending to remove it once the new setup proved stable and available.

About three hours later, the exposed database appeared to have been discovered and deleted by an automated ransomware script. clay said that when he saw the error message on his phone, he understood that this was tantamount to receiving a disaster alert.

The error message contained the keyword “drop”, the SQL command used to delete the database.

When he then logged into the MongoDB machine to check what state the database was in, he found a new empty database named “READ__ME_TO_RECOVER_YOUR_DATA”. It contained a message left by the hacker: “Your data has been deleted, you must pay 0.03 BTC within 48 hours to recover it.”

Clay quickly looked at the MongoDB access logs. Analyzing the logs revealed two connections that were made a few seconds apart. Both of these connections from unknown IPs occurred just before the entire database was deleted. By tracing the connection IPs, it was easy to spot the hacker entering the server and deleting it just seconds later.

But in fact, Clay enabled the UFW firewall and tried to stay on a strict permission list, only for internal servers. But for Docker reasons, it didn’t work on the new server.

When he containerized MongoDB, Docker inserted a permissive rule in iptables that opened MongoDB to the world, so when the firewall was “active”, running sudo iptables -L | grep 27017 showed MongoDB. grep 27017 to show that MongoDB is open.

Fortunately, Clay did not pay the ransom to the hacker, because through analysis he believed that the data was not actually stolen and that he had a backup (copy) of the wiped database. The logs also show that the automated attacker only issued a delete command and did not transfer any data from the MongoDB server.

Another evidence that the data was not transferred is a snapshot of the traffic on the db-mongo1 machine within 24 hours.

6:10pm: The new db-mongo1 server is rotated into the MongoDB master server. This machine is the first new, soon-to-be-private cloud machine.

9:35pm: Three hours later, an automated hack attempted to open a connection to the db-mongo1 server, and the database was immediately dropped. Downtime ensues.

10:15pm: Before the previous primary server went into rotation, NewsBlur took a snapshot of the server to ensure that the backup would not self-delete upon reconnection. This process took a few hours of downtime, but saved almost 18 hours of data by not forcing me into a daily backup archive.

3am: Snapshot completed and replication from the original master server to the new db-mongo1 begins.

4:30am: Inbound replication from the old primary server is complete and outbound replication begins on the new secondary server. NewsBlur is now back to normal.

It took over 3 hours from the time the switch was flipped to the time the hacker deleted NewsBlur’s MongoDB collection and pretended to redeem about 250GB of data. A few hours later, when the backups were restored, NewsBlur was back online and the vulnerabilities created by Docker were patched.

What can I do to make sure this doesn’t happen again?

For the past seven years, “Docker networks bypassing firewalls with no option to disable” has been a concern for developers. It was reported to Docker back in 2014, but the problem was never resolved.

While there are various suggestions for solutions in online forums, Clay commented on why Docker didn’t choose to just fix it: “It’s a bit of a trade-off between convenience and security. Here, Docker thinks convenience is more important.”

On the other hand, attacks on MongoDB databases in an open state have existed for a long time. After this incident drew widespread developer attention on Riddit, noted security researcher Bob Diachenko tweeted.

Notably, the MongoDB was insecure online for nearly a month before the “hackers” cleaned it up.

Professionals believe the attack was automated and part of a ransom trend that began in late 2016. At that time, multiple hacking groups began wiping MongoDB servers and demanding ransom from server owners who had exposed MongoDB instances to the network without passwords. While the hackers claimed they had copied the deleted data, in most cases it was a bluff, expecting victims to pay a small ransom, with ransom demands typically ranging from $200 to $2,000.

These attacks have been around for years and are still active today. The lessons we can learn from this incident are.

  • Migrating data to a VPC, or Virtual Private Cloud, allows critical servers to be accessed only through other servers in a private network. But it was this migration process that was subject to hacking. Ironically, when migrating data to a private network, NewsBlur needs to migrate all data from publicly accessible machines. It is the first step of the process.
  • Database user authentication mechanism needs to be enabled on all databases. For a long time, NewsBlur relied on firewalls to protect itself from threats, but when the firewalls silently failed, the database was exposed. Today, who can say that if the firewall fails, but because the authentication mechanism already exists, then the hack will be detected. This password needs to be long enough to avoid brute-force cracking, because at the end of the day, an open database that is not protected by a password is likely to be on the list of hackers.
  • Database users with delete privileges need to be tuned. Most database users should only need read and write permissions. Ideally, only localhost users should be allowed to perform potentially destructive operations.

But these are only part of a defense strategy, and a proper defense strategy must not rely on a carefully set up layer alone. For NewsBlur, this layer is a firewall that only allows access.

As always, what’s really useful is a “backup” policy. Regular, tested backups are a necessary part of any web service.

Reference link.

http://blog.newsblur.com/2021/06/28/story-of-a-hacking/

Reference https://www.infoq.cn/article/QQYuiOLc8xR2zzvdQowb