What is log4shell?

The Log4Shell problem is a security problem that was discovered in Java software.
The problem affects servers in the world around the world.
Since December 10, 2021, days after industry experts discovered the Log4Shell flaw in servers supporting the Minecraft game.

According to one team tracking the impact.
The vulnerability is a potential threat to millions of other applications and devices worldwide.

In the days following the announcement of the vulnerability, Check Point tracked millions of attacks initiated by people, with some research studies at a rate of over a hundred attacks per minute, which ultimately resulted in over 40% of the world's business networks being attacked.

Cloudflare CEO Matthew Prince says evidence of use or apparent testing of the vulnerability was seen as early as December 1, nine days before the vulnerability was publicly disclosed.

Why We Closed Our Minecraft Servers Immediately?

Our security system and some functions in the Minecraft game were at risk from abuse and damage to our quality and security capabilities.
Minecraft is the source where the vulnerability was discovered.

The Aftermath

We had to update and change a lot of the source code for our in-game security system
and manually update 100k files to meet the new security system configurations.
while some of the poses were automatic we still had to do a lot of work by hand
also, all other plugins and servers had to be updated to the new server software that has been released to fix log4shell

Long Days

me and our small server team, worked around the clock for a week to fix the issue on all servers and update over 200K lines of code, 200+ plugins, and over 500K configuration files.
This is not even counting the sheer number of hours spent on testing the security of the new system and figuring out different compatibility issues.
As a mature DevOps engineer, it was kind of my first major skill check.