Friday, April 24, 2020 - 6:55 PM

I was at the grocery store when I got a Blocknative notification that a new contract was being deployed. Remember that this was right at the peak of the COVID-19 pandemic, so this was my first shopping trip in 10 days, and it’s intended to last me at least another week. I find a corner of the store where I can be out of the way and jump on a call with the engineering team.

While we were confident that our deployment from Wednesday would prevent the attacks we’d already seen from causing any issues, there were two reasons we still wanted to stay ahead of the attacker:

1. They might hit us with a completely different attack. We had already identified a handful of other attack vectors they could potentially use against us, and we were still in the process of putting mitigations in place to prevent those.
2. We wanted their IP address. We were only collecting IP addresses of calls to a finite set of contracts; if we wanted to collect theirs, these new contracts needed to be in that set.

Normally I would be the one to implement these kinds of changes, but I was at the grocery store. I talked another engineer through the changes, he sent me the code to review on my phone, and we pushed the changes out just minutes after the block confirmed.

I hurried through the rest of my shopping trip and headed home to check the logs. Sure enough, we had an IP address. I plugged it into Google to find out what part of the world it was from.

I was surprised by the result. I had expected to get nothing more than a WHOIS result ¹. What I found instead was a page managed by the Syracuse University Full Stack Security Lab. The page had instructions for testing a particular hardware security measure, had instructions for students to log into the exact IP address that was attacking us, with directions to get credentials from Dr. Yuzhe Tang, who is in charge of the lab.

The computer that was attacking us belonged to a University’s security research lab. Now this was interesting. I have a master’s degree in Computer Science. I worked in a security research lab at the University of Tulsa from 2007 through 2010. Based on my experience, I now assumed there were two possibilities for why we were being attacked by this computer:

1. The computer that was intended to be used for coursework had been compromised by third party hackers. I had found credentials for the server with just a few minutes of effort. It was ostensibly only available from within the Syracuse network, but it’s not hard to imagine that someone had chained attacks together to get access to Syracuse’s network then use this machine to carry out an attack.
2. Some bored student had an idea for a DoS attack and decided to try it out. It’s certainly not unheard of for smart, security minded students to try out hacking without understanding the legal and ethical ramifications of their actions.

This changed our strategy. If it was the student, they were relatable. I had an internship in college where I’d been employed as a penetration tester for a local software company; if I hadn’t had that as an outlet for my own interest in hacking I might have done similar things myself. Another member of our engineering team did some things when he was younger where he was lucky he didn’t get caught.

If it was a student, we wanted to talk to this kid. We could help them understand the risks they were taking and the problems they were causing, and if they were interested in continuing we could give them access to our testing environment where they could explore, we could find ways to improve our own service, and our production systems wouldn’t be threatened by their actions.

We changed the error message associated with the attacker’s contracts to an invitation. The message said that we wanted to talk, and gave them a Google Meet link to join us on a video call. We also included the email address of the professor in charge of the research lab that computer belonged to — a clue that we were onto them.

Then we waited. After an hour, when nobody joined our call we started exploring other options. We decided to route any requests from the attacker’s IP address to our testing environment and open it up to attacks. This would let us see what new ideas they came up with without putting our production servers at risk ². Still, nothing came.

As it appeared that no security research students would be joining our call, we decided to wrap up for the night. Before I went to bed, I sent an email to the professor in charge of the research lab, copying the dean in charge of the computer science program, and also copied the University’s IT department. I gave them a quick summary of the attacks we had seen over the week, gave them the IP address we had tracked things down to, and asked them to please ensure that no further attacks come from their network. If the attacker was a student, I expected things would stop after this; if the attacker were a third party using a compromised lab computer for their attacks, I figured they’d change IPs after that.

What actually happened the next day was something I never would have imagined.

Continue the story on Saturday