Skip to main content Link Search Menu Expand Document (external link)
LIS 640

Assignment: Generative AI and Cyber

Your goal for this assignment is to think about how recent advances in generative AI (ChatGPT, Copilot, DALL-E) might affect cybersecurity.

For this assignment, you need to experiment with some new generative AI tool, and think about ways that having it (or something like it) might change the way cybersecurity is done.

As you think about this, there are two major categories of changes that might happen:

  1. Malicious attackers might use generative AI in some way that makes their attacks better in some way (more effective, harder to track, scale up better, etc.).
    OR
  2. cyber defenders might be able to use generative AI in some way that helps them protect against attacks in some way (detecting abnormal patterns; developing more realistic honeypots to identify and catch attackers; initial screening of alerts; easier information sharing across organizations; better scanning for bugs in software, etc.)

You don’t need to think about ALL of the ways that generative AI might affect cybersecurity. Choose one specific way that you think generative AI will change cyber – a way that is currently being used, or a way that might be able to be used in the near future.

It is important that this be realistic; it is NOT OK to envision future AI systems with new capabilities beyond what current AI systems can do (e.g. Artificial General Intelligence or AGI). Instead, I highly recommend that you spend some time with current AI systems and look at what they can and cannot do, and how well they currently work. The University of Wisconsin provides all of us access to the latest AI tools from Microsoft, called Microsoft Copilot. You can sign into Copilot and use it to access the GPT-4 large language model (the same model powering ChatGPT pro), and also (by asking it to generate images) the DALL-E 3 image creation AI tool. You can give it documents, text, or images to summarize or analyze, and you can ask it to generate new creations. It has access to current events. Also, if you access it using the university-provided login, then commercial data protection is enabled and none of what you give it will be retained by Microsoft or OpenAI and it will not be used to train future models.

Think about Motivations

Once you think about the technology and how it might be used, I’d like you to go two steps farther.

First, think about the motivations of the person, or people, who will use it. WHY would they do this? Would this help them achieve some goals? If so, which goals, and how will those goals influence the way that AI is used? Are there simpler alternatives?

For example, one possible use I’ve seen proposed for generative AI is phishing; malicious hackers can use generative AI to generate large quantities of realistic emails or text messages that try to get you to click on a link. Technically, this is true – this is exactly the kind of thing generative AI is good at. Now, let’s look at the motivations of the attackers. Why would an attacker do this? It isn’t because phishing is the end goal; they likely want to steal credentials and sell those credentials. Is there an easier way to generate those emails? Yes, yes there is; it is really easy to find legitimate emails and then just copy those emails. Which is more realistic – a ChatGPT generated email pretending to be from IT, or an actual, legitimate email from IT with the URLs changed? Is ChatGPT really going to help that much here? If ChatGPT costs money to get a decent version of it, would it be worthwhile for attackers to pay for it, or would they just do things an easier/cheaper way?

Now, once you realize that, you should start to think about what advantages ChatGPT actually has. It isn’t just that it can generate a fake email; that’s easy. It can (somewhat) intelligently go back and forth. So, instead of thinking of phishing as a single fake message, what if you think of something like a romance scam where an attacker pretends to be a potential boyfriend/girlfriend and sends lots of messages back and forth before actually stealing money. Right now, romance scams are really dangerous to victims, but hard to do at scale because it takes a lot of work for an attacker to pretend to be a boyfriend/girlfriend. Could ChatGPT automate some of the work of responding regularly, so that an attacker could do hundreds of people at the same time? I don’t know; try it out and see?

That is the kind of thinking I want you to do. Think about the motivations of the attacker or defender, what they are trying to accomplish, and what ways that generative AI might help them, and whether there are other ways to do that same thing easier or cheaper. Hopefully that thinking will help you think more carefully about the capabilities of generative AI.

Think about the arms race

Second, remember that cybersecurity is always an arms race. Every time one side has a new innovation, the other side reacts and changes the way they do things. If an attacker uses generative AI to do something new, then defenders will try to change things to defend against that new attack. If defenders use generative AI to better defend their systems, then attackers will adapt and try to find new ways around those new defenses.

For this assignment, I want you to think through at least one next round of the arms race. If attackers/defenders adopt your proposed use of generative AI and it works, how is the other side likely to react? Is there something that they can do to make it not work very well? Is there something they can do to detect and block the AI?

For example, if attackers start using ChatGPT to generate phishing emails, are there things that defenders can do to stop this from working well? Could OpenAI modify ChatGPT to refuse to help people with emails? (probably not a good idea; there are many legitimate reasons to use ChatGPT to help with legitimate emails…) Could ChatGPT watermark emails so that it is obvious when it was used? (If so, would that catch lots of real, legitimate emails that ChatGPT helped with?) What else could defenders do to try to stop this?

Report

Above, I describe three steps of work you’ll need to do for this assignment:

  1. Investigate generative AI and identify at least one way that either attackers or defenders can use it to do their work better
  2. Think about motivations – why would those people actually want to use generative AI? Would it really help them?
  3. Think about reactions in the arms race – how will the other side react when the first side starts using generative AI in this way?

These steps aren’t linear. As you think about each one, you will gain new insights and ideas about the others. Thinking about motivations might lead you to change your idea of how generative AI will work. Thinking about reactions might lead you to conclude that it won’t actually work, and lead you to think about a new, different way to use it. Think about how AI works and what it can be used for, you might realize new possible defenses.

Once you’ve done this thinking, it is time to write up a short report. Write up your vision – what you think will happen (or, is currently happening) and describe it in details. Be sure to describe thinking from all three parts of the assignment – generative AI uses, motivations for use, and reactions of the other side.

I strongly encourage you to use the report to make an argument. Likely, the argument will be some form of “this is coming and we should be prepared for it”. On the other hand, if after thinking things through, you decide it isn’t a big deal, the argument might be “the generative AI hype is overblown and likely won’t affect cyber in this way because X, Y, and Z”. Either argument is OK, as long as it is well-supported by careful thinking.

I also strongly encourage you to provide detailed examples to support your points. For example, if you are going to argue that ChatGPT / Copilot can generate good phishing emails, it would help to have at least one or two phising emails that have been generated by ChatGPT / Copilot. If you are going to argue that these tools can detect phishing emails, then show a transcript of it actually doing this.

Alternatively, another type of evidence could be research papers and/or news articles of people actually using generative AI to do whatever it is you are proposing. That’s OK too.

Presentation

The report is due on April 11 (Thursday), before class. In class that day, you will be asked to give a presentation to the class about your ideas for generative AI. The presentations will be limited to less than 5 minutes – they don’t have to be that long, but they can’t go over. Your presentation should discuss all three aspects of the report (use, motivation, reaction) and also should convey the argument you are trying to make in your report. Try to convince your fellow classmates!

You do not have to use slides, but you are welcome to. I encourage you to include examples from ChatGPT / Copilot / etc. in your presentation (either as screenshots or as a live demo).

Individual or Pair?

For this assignment, you have the option of doing it either individually (by yourself), or by working with one other person as a pair. If you work as a pair, you both should turn in exactly the same report with both of your names on it. It is up to you how you break up the work, but you both will be responsible for answering questions and demonstrating that you understand the ideas in your report during the presentation.