Summary
In this episode of the AI Engineering Podcast Kasimir Schulz, Director of Security Research at HiddenLayer, talks about the complexities and security challenges in AI and machine learning models. Kasimir explains the concept of shadow genes and shadow logic, which involve identifying common subgraphs within neural networks to understand model ancestry and potential vulnerabilities, and emphasizes the importance of understanding the attack surface in AI integrations, scanning models for security threats, and evolving awareness in AI security practices to mitigate risks in deploying AI systems.
Announcements
Parting Question
In this episode of the AI Engineering Podcast Kasimir Schulz, Director of Security Research at HiddenLayer, talks about the complexities and security challenges in AI and machine learning models. Kasimir explains the concept of shadow genes and shadow logic, which involve identifying common subgraphs within neural networks to understand model ancestry and potential vulnerabilities, and emphasizes the importance of understanding the attack surface in AI integrations, scanning models for security threats, and evolving awareness in AI security practices to mitigate risks in deploying AI systems.
Announcements
- Hello and welcome to the AI Engineering Podcast, your guide to the fast-moving world of building scalable and maintainable AI systems
- Your host is Tobias Macey and today I'm interviewing Kasimir Schulz about the relationships between the various models on the market and how that information helps with selecting and protecting models for your applications
- Introduction
- How did you get involved in machine learning?
- Can you start by outlining the current state of the threat landscape for ML and AI systems?
- What are the main areas of overlap in risk profiles between prediction/classification and generative models? (primarily from an attack surface/methodology perspective)
- What are the significant points of divergence?
- What are some of the categories of potential damages that can be created through the deployment of compromised models?
- How does the landscape of foundation models introduce new challenges around supply chain security for organizations building with AI?
- You recently published your findings on the potential to inject subgraphs into model architectures that are invisible during normal operation of the model. Along with that you wrote about the subgraphs that are shared between different classes of models. What are the key learnings that you would like to highlight from that research?
- What action items can organizations and engineering teams take in light of that information?
- Platforms like HuggingFace offer numerous variations of popular models with variations around quantization, various levels of finetuning, model distillation, etc. That is obviously a benefit to knowledge sharing and ease of access, but how does that exacerbate the potential threat in the face of backdoored models?
- Beyond explicit backdoors in model architectures, there are numerous attack vectors to generative models in the form of prompt injection, "jailbreaking" of system prompts, etc. How does the knowledge of model ancestry help with identifying and mitigating risks from that class of threat?
- A common response to that threat is the introduction of model guardrails with pre- and post-filtering of prompts and responses. How can that approach help to address the potential threat of backdoored models as well?
- For a malicious actor that develops one of these attacks, what is the vector for introducing the compromised model into an organization?
- Once that model is in use, what are the possible means by which the malicious actor can detect its presence for purposes of exploitation?
- What are the most interesting, innovative, or unexpected ways that you have seen the information about model ancestry used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on ShadowLogic/ShadowGenes?
- What are some of the other means by which the operation of ML and AI systems introduce attack vectors to organizations running them?
Parting Question
- From your perspective, what are the biggest gaps in tooling, technology, or training for AI systems today?
- Thank you for listening! Don't forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email hosts@aiengineeringpodcast.com with your story.
- To help other people find the show please leave a review on iTunes and tell your friends and co-workers.
- HiddenLayer
- Zero-Day Vulnerability
- MCP Blog Post
- Python Pickle Object Serialization
- SafeTensors
- Deepseek
- Huggingface Transformers
- KROP == Knowledge Return Oriented Prompting
- XKCD "Little Bobby Tables"
- OWASP Top 10 For LLMs
- CVE AI Systems Working Group
- Refusal Vector Ablation
- Foundation Model
- ShadowLogic
- ShadowGenes
- Bytecode
- ResNet == Resideual Neural Network
- YOLO == You Only Look Once
- Netron
- BERT
- RoBERTA
- Shodan
- CTF == Capture The Flag
- Titan Bedrock Image Generator
[00:00:05]
Tobias Macey:
Hello, and welcome to the AI Engineering podcast, your guide to the fast moving world of building scalable and maintainable AI systems. Your host is Tobias Macy, and today, I'm interviewing Casimir Schultz about the relationships between the various models on the market and how that information how that information helps with selecting and protecting models for your applications. So, Kasimir, can you start by introducing yourself? Yeah. Hi. My name is Kasimir Schultz. I'm the director of security research at Hidden Layer. And do you remember how you first got started working in ML and AI? It's actually been a interesting journey. So I started with a more traditional security background. And, actually, even before that, more in the software engineering side of things. And,
[00:00:49] Kasimir Schulz:
I was doing traditional security zero day research to try and find vulnerabilities and just anything out there. And I came across Hidden Layer, and I actually initially joined to do security in just anything AI related. So the AI frameworks, AI model formats, and things like that. And, once I joined Hidden Layer, I kinda went down the rabbit hole machine learning, and I've been learning from our data scientists, learning from just breaking things and reverse engineering things. So it's been a fun journey there. Yeah. It's definitely a very important area of research even before the current epoch of all of the generative AI systems just because of the fact that there is so much potential for issues
[00:01:28] Tobias Macey:
and attacks because of the probabilistic nature of the software where you can't just run a static analyzer and say, oh, that's where the problem is. That's what I need to patch. You have to worry about so much of the overall process that goes into building these models and then operating them and maintaining them because it's not just a single application or a single process that you need to worry about. It's all of the data that goes into it, all of the code that goes into building it, and all of the process around actually keeping it secure in in its actual runtime.
[00:01:58] Kasimir Schulz:
And, I mean, I'm seeing that a lot more now as well. So one of the biggest issues I'm seeing nowadays is just all of this agentic stuff. People are just trying to innovate, which is awesome. There's so much cool stuff around that. But the issue with that is people aren't really considering what goes into it and what does the model have access to. So people aren't considering what the model actually has access to and then the frameworks that it can interact with. For example, I was just working and published a blog on MCP. So the model context protocol last week. It's just been a hot topic. Everyone was talking about it. I was at a conference last week and I had great discussions with people about it as well. And one of the things that we noticed is that a lot of traditional security vulnerabilities are in these new formats and these new agentic systems. And then a lot of these just agentic systems also have these innate systematic vulnerabilities and weaknesses as well. So for MCP, one of the things that we discovered is that a lot of the MCP servers that were written and that were out there and recommended by people, they allowed code execution.
And the reason that they allowed code execution is because they were running on your system. Right? So if you're connected to cloud, you're working through it, everything's fine, nothing bad should happen. And people are writing it along the context of that's the only attack plane that there actually is. However, once you start integrating more MCP servers, you have a lot more attack surface because, for example, reading a file on your local file system. What do you do if there's an indirect prompt injection in there that then tells you to run, arbitrary code with another MCP server? Or for example, we were using the Ghidra and binary ninja MCP server to reverse engineer binary.
And in that malicious binary, so the malware, there was a prompt injection in there, and it started executing arbitrary code on our system as well. And even if you don't have arbitrary code execution, just kind of the mesh of all of these different tools that agents have nowadays allows for just a whole new area of attack surface. So one of the things that MCP does is and MCP cloud, desktop is you have all the permission management. So you can allow a tool to run once or you can allow it to run for the session. And then every new tool, you have to give permissions. However, once you're actually running through a tool, sometimes one single prompt from you requires the same tool be to be called, you know, fifteen, twenty times. So you're gonna click allow for the session. So what we're able to do was just by asking Claude to download and summarize a document for me, because it downloaded the document, it used fetch. And then because it downloaded and read the document, it used the file system tools. And then at that point, there was a prompt injection in there that actually let us read all of the file system and then exfiltrate the data since we already have those tool permissions.
[00:04:39] Tobias Macey:
Yeah. That's definitely it's definitely a combinatorial explosion of attack surface by virtue of all of these integrations that you're trying to put together where, up until now, integration was a task in and of itself, and so it required a lot more care and intention behind it. Whereas now you're handing a lot of that over to these black box systems that you don't have full visibility into what they're doing or what they might be doing. And it's all in the name of convenience, and that that's always the challenge with security is how much convenience to allow for and how much to lock things down. So the more secure it is, the harder it is to use oftentimes unless the security was baked in from the beginning, which it's sounding like was not really the case with these MCP servers.
[00:05:25] Kasimir Schulz:
Yeah. Exactly. But it's a old cat and mouse game that's been around for a long time in security, so we'll get there. Absolutely. So broadening the scope a little bit more, I'm wondering if you can give some overview about the current
[00:05:39] Tobias Macey:
state of the threat landscape for ML and AI systems beyond just this model context protocol that has taken the attention for the past couple of weeks?
[00:05:48] Kasimir Schulz:
Yeah. Of course. So when I first got into the field, I told you I came from a traditional security background. So one of my first task was kinda flushing out all of the low hanging fruit. And while the field definitely has done a much better job at getting more secure in a traditional security standpoint, for example, I think it was maybe a month or two ago. Time's been flying like crazy. But, PyTorch now does Torch load safe by default. So for over a year and a half that I've been working at hidden layer, that was one of the things that we kept seeing. There were over 500,000 projects on GitHub that had potential code execution just because they were loading a machine learning model. And we're seeing a lot of that not just with, necessarily the models themselves.
At this point, I feel like most people know a lot of these models have pickle inside and know to kinda use them safely, but we're seeing these with the safe formats as well. So everyone assumes safe tensors is safe. However, just because the file itself doesn't execute code, doesn't mean that just by loading it, itself doesn't execute code doesn't mean that just by loading it, you might not execute code. So with DeepSeek and all of these new architectures that are coming out, you have to have arbitrary code or trust remote code on when you load them through Hugging Face transformers. And what a lot of people will do is they'll look through the code. The code will look fine, especially because a lot of times what happens is that they copy the code directly from the transformers repository, just make whatever changes they have to. However, by using transformers to load models, you can still have arbitrary code executed that's not actually in the repository that you're looking at. That's this repository side loading that we're seeing a lot more. There's actually a few instances of it out on Hugging Face. And pretty much what happens there is if you have trust remote code on for Hugging Face, transformers and you're loading in a safe tensors file, it'll look at the config dot JSON. And then that config dot JSON, it'll map what classes are stored in what files. However, it can also point at other repositories.
So you might be thinking that the code that you're running is the deep secret repository that you're looking at, but it's actually running some other malicious code instead. So that's definitely one of the big things we've seen. And then we've been seeing a lot more true AI vulnerabilities and AI attacks lately, which has been really exciting to see, just kinda what are people coming up with, not just in the sense of generative AI, but also predictive and classification models.
[00:08:10] Tobias Macey:
And on that point of the divide between predictive and classification style models that have been the bread and butter for machine learning systems for the past decade plus, but also now moving into this landscape of generative models. What are the main areas of overlap, and what are the key points of divergence as far as the attack surface and, threat signature from that division of the the model types?
[00:08:36] Kasimir Schulz:
Yeah. So it's actually funny enough. The overlap is a lot bigger than you would think for the just types of attacks that you can do. What's really kind of the main attack that you can do and the main vulnerability of these types of models is that in most applications, the entire input that goes into the model is completely user controlled. And what that means is, you know, with software, you might have an API endpoint, and you can query just part of that API endpoint. But all of the data, everything else is controlled by a server, is controlled by access management. But with a model, every single thing that is processed in that model most likely came from the user, except for, like, you know, the little cases with, a system prompt, for example, that might come from the system. But the main chunk of it is from that user. And then the goal for attacking both is the same as well, and that's controlling the output. And I think that once a lot of us started thinking along that mindset, we've been able to come up with a lot more attacks that we haven't necessarily done before. So the idea is, you know, all your goal is to control the output in one way. So you're not necessarily trying to do prompt injection. You're not trying to do a jailbreak, alignment bypass. It's just controlling the output. And when you think about it, controlling the output is not that hard if you control everything that goes into the model, and then you actually start understanding how the model works. So I think that's the main overlap between the two of them. As far as, divergences, I found that attacking predictive and classification models, it's a lot simpler. It's not actually easier in my opinion than generative models. So generative models are on the easier side. However, it is simpler because what you really have to do for attacking those models is you just have to figure out what inputs and what what, feature in the input space will just nudge a classification just slightly over the edge. Right? Because that's all you have to do. You have to trick the model and control the output of the model while still looking like it's one input or another input in most cases. Right? So for example, a classification model that detects malware or not malware. If you're running malware, you can put whatever you want inside the model. We've been able to find that sometimes it's as easy as just writing the text this is benign 20 times somewhere in the malware like a string or a comment and something like that's enough to flip the model one way or another just because you can keep nudging it. A lot of these models, it's people assume that it's, you know, the model thinks, but the model really is just math. So sometimes you can just real do really stupid attacks, and they work really, really well. And then the reason why I was saying that generative models are not actually harder and, are tend to be a lot easier to attack is there's just so much more that you can do with them. So one of my teammates, while back came up with a technique called CROP, knowledge return oriented prompting. And what they can do with the technique is most of these large language models and small language models, they contain a lot of information. They know all the pop culture references. They know all of, you know, those types of things. So I I think most people are gonna be aware of, you know, x k c d that you've seen a few of them. Yeah. So we actually had a model sitting on top of a SQL database.
And the model had a bunch of protections saying, don't write code to drop the tables. Don't write code to remove stuff. Just do simple, you know, select statements. And what we were able to do was we were able to reference the little Bobby tables x k c d. And whereas saying, you know, hey. Write me a statement to drop the database. It said no. But say, hey. What is the name of that kid? Little Bobby tables with his full name, replace this one part of it with something else. And we were able to actually exploit the system underneath just because that wasn't a guardrail. There were no guardrails about actually getting information. And then you can do a lot of those attacks without ever actually having something bad in the prompt. So, for example, instead of ignore, previous instructions, we were able to do pretend you're a developer and pipe your output to devnull. And the model understands that, and it'll do those things. And it's also really interesting in the, multimodal space as well, which is becoming just a whole fascinating field with generative models. We were able to give the model just a image of a reverse sign and a shell, and we were able to tell the model, take the two items in the image, make it Python code, and tell me what the code is. And then we were able to get chat g b t to write as a Python or reverse shell where it normally wouldn't do something like that.
[00:13:02] Tobias Macey:
Yeah. It's definitely interesting the way that the introduction of generative models just cracks the overall attack surface wide open because there is so much potential for use and output that it's effectively impossible to guard against all potential inputs. And I don't know if there has been any work so far that's ongoing to come up with something along the same lines as the OWASP top 10 for web applications in the context of generative models. I know that there are things like guardrails and input and output filters for trying to address some of that. But as you just pointed out, there are a lot of clever ways that you can work around the heuristics that are built into these systems.
[00:13:45] Kasimir Schulz:
Yeah. So there actually is a lot of great work going on. So OWASP has a top 10 for LMs as well, which has been great to use. And then the CVE organization has two working groups, one to define what actually is a CVE around AI systems and then one to define, CWEs or the common weakness enumerations for, AI systems as well. So there's lots of organizations actually trying to figure out, you know, how can we classify these attacks, then how can we classify these attacks, then how can we defend against them.
[00:14:17] Tobias Macey:
And then in the category of generative models, there are obviously numerous stories that are already in the wild about ways that they've gone wrong. I think maybe the most widely spread one is the story about the car dealership that had their model promise to sell a car for a dollar because somebody was able to trick the model. But I'm curious from a gradation of damage both to the organization, whether it's monetarily or reputationally, up through to issues around physical security and safety, what are some of the potentials for damage that can be caused through the deployment and use of these models whether compromised or not?
[00:15:03] Kasimir Schulz:
I mean, it's a big question. There's a lot. So, obviously, you know, the situations with the new hallucinations or prompt inject it. Like, you, mentioned there's brand issues, you know, if your model suddenly makes a bad joke about your brand and it's your chatbot on your website, that's not great either. With a compromised model, I mean, now that we're hooking them up to agentic systems, that's definitely now there's actual real world impacts of, you know, the system doesn't just tell you something. It actually does an action for you. So it might delete your database or it might run malware on your system. And then there's also a whole interesting kinda part now where we're seeing a lot of these, jailbreaks, especially for local models. And what's really scary there is, as I mentioned earlier, a lot of these models have a lot of information. Right? They know all those pop culture references. They know how to do specific tasks, and they they have all that information that was fed into them to train them. And they're trained so that they won't tell you how to make a bomb, won't tell you how to make make malware. But with local models, they're jailbreakable.
People can either fine tune those guardrails out of them, or there's, new work being done called refusal vector ablation. It's a really cool paper a while back where they realized that the refusal direction for a model was pretty much mediated by one single vector in the model. And what you could do is you could just ablate that, vector. So I actually have five three, five four, llama three, and gemma all fully jailbroken on a local machine, done on a CPU. So I was able to, you know, completely remove those guardrails without a fancy GPU, actually, on the computer I'm speaking to you on today. But what's kind of scary about those things is that even five three where it was trained on synthetic data, it's still able to answer how to make a bomb, how to make math, things like that. And now, because it's on my local system, those searches are completely untraceable. It's not like, you know, asking chat g b t how to make a bomb like the person who did with the cyber truck explosion a few months ago in Vegas. It's not like Google search for somebody can monitor that. So this is all just completely unmonitored on every anyone's laptop now.
[00:17:14] Tobias Macey:
Yeah. That's definitely a scary scenario. The availability of information by virtue of it having been compressed into these models, to your point, without any sort of network requirement to actually go out and fetch that information. Yeah. The other interesting shift in terms of the AI landscape that was brought in by the rise of generative models is the widespread use now of foundation models as a building block for systems that are being put into production and the fact that most of those foundation models are originating from a small handful of organizations and companies.
And I'm wondering what you see as the level of risk both in terms of platform risk from a organizational perspective, but more specifically, security risk of these foundation models being distributed through systems such as Hugging Face and the fact that so many organizations are relying on a small pool of models to then go and build a wide breadth of services on top of?
[00:18:16] Kasimir Schulz:
Yeah. So a lot of smaller companies don't have the ability to create those foundational models. So foundational models definitely are, you know, something that's useful, something that's good for just the majority of people. However, because a lot of them are so similar, a lot of the same techniques work throughout all of them. But because they're managed by different organizations, we tend to have to report pretty much the exact same issue to every single model provider. And the reason for that is they don't talk to each other as much just because I mean, why would you in such a situation? Right? And we're actually gonna be publishing some research next week, where we have a single prompt that works on every single foundational model, and can be used to either jailbreak or leak the system prompt. So definitely is a bigger supply chain issue than people might think about.
[00:19:04] Tobias Macey:
On that note of the commonalities between these models, you recently published some work that you've been doing at Hidden Layer around the shadow gene where you're figuring out what are the common subgraphs within these model structures and these neural networks that are shared and reused across different families of models and probably even across families just because it's such a a common pattern in deep learning and neural network design. I'm wondering if you can talk to some of the ways that that information about the shared subgraphs informs the potentials for bad actors to identify models to target for these types of attacks that are known to work against one particular logic path.
[00:19:47] Kasimir Schulz:
Yeah. Before I go into ShadowJeans, though, I do wanna talk a bit about Shadow Logic, which is what we published right before and which is actually what led us to ShadowJeans. So I think last June ish, I had started working on a lot of these new backdoors that were coming out. So the backdoors were poison datasets, backdoors that were fine tuned into models. And one of the biggest issues I saw was that if you fine tuned one of these models again, a lot of times the backdoor would disappear just because they were put in there through the fine tune. And I wanted something that would persist across the fine tunes, persist across model conversions. And the reason for that is because a lot of times, as you said, somebody might take a model off of Hugging Face, but then they might adjust it slightly for their use case. And that's when I actually looked into it a bit more and found out about the computational graphs. So the computational graphs are I like saying that the weights and biases are almost like the memories of the model, and then the computational graph is like the synapses in the brain. So it's actually how the input traverses the model, how each operation gets applied, how the weights and biases get applied, and then finally, we get to the output. And normally, these are just a lot of mathematical operations as well as, you know, just the more AI mathematical operations. So like convolutions and sigmoids and stuff like that. But what we were able to do was because it really is just a pathway of mathematical operations.
To me, it felt a lot like traditional byte code, an assembly code. And I was able to start coding in any actual logic I wanted. So I was able to code in logic for a Resnick, image classifier where if a pure red pixel, so two fifty five zero zero, was present anywhere in the photo, it would change the class from one class to another. I was able to do it with a YOLO model. So a lot of YOLO models are in, security cameras. We actually hacked a security camera a while back that actually contained the YOLO model in it. And by lifting up a cup, so if a cup and a person were present in the same time, the person would disappear. So you could walk up to the camera and you'd be completely invisible, but only if you held a cup or only if you had a cup and glasses on. We were able to do it with generative models as well. So with five three, have it where the model would respond completely normally except when you knew a specific passphrase. And then you could give the passphrase in whatever output you wanted, and it would just output the actual model that you wanted. But while looking through all of these, we were spending days way too much time looking at the computational graphs through Metron, which is a great open source viewer for these graphs.
And looking through them, we started realizing that we were recognizing what the actual models were and what families they were coming from. So at the start, we had to, you know, see, oh, what model is this? Where is something? And by the end, one of us sent one of the other people a new model. We could immediately tell, oh, that's a BERT model or that's a resident model. And, even if it was a model that we'd never seen before, like, if we had never seen Roberta before, we were able to tell, oh, that's where the transformers is. That's what I need to change there. And that's what led us to the idea of shadow genes. So what we realized was we were recognizing these repeated subgraphs inside of the models because they were the same throughout all these other models. So even if the model had slight modifications, if you wanted one specific flare to it, you still have this core components.
And what we were able to do with ShadowJeans is we were able to write signatures. So the signatures match a specific subgraph inside the greater graph. And with these signatures, we can tell if a model is a specific family, what components are in the model, and then even if the model was derived from another model. So going back to the Roberta example, if we had never seen Roberta before, we had never written a signature for Roberta. We have seen BERT before. So we would have been able to tell you that the Roberta model was derived from the BERT model even if we'd never seen it before. And what's really cool about that, the attack space, which is what you're asking about, is that a lot of these models have certain attacks, not vulnerabilities, but attacks that kinda transfer between the models.
So, for example, the security camera that we hacked a while back, it was, one of the Wyze security cameras, which are these budget home cameras, and they have the AI model actually on the device now. But going through the model, we could see that they had a fine tuned version of YOLO there. It was their own model format too, so it was all wonky, and it was quantized as well. But because we could tell it was a YOLO model, we were able to generate a bunch of attacks on our computers with a standard YOLO model, and about 20 of the successful attacks transferred to the security camera. So we were able to train on something where we actually had the ability to scale up GPUs instead of having to send one image through the camera, which took maybe ten seconds to process each time.
[00:24:50] Tobias Macey:
Bringing us back around to the supply chain vectors, Once you have a model, you're aware of the different subgraphs. You're able to add in some of these alternate logic paths because of the fact that you know that, oh, hey. This is doing this mathematical computation. And if I want it to go this other way in this subset of use cases, I can add in an additional graph to the neural network. And then once you have that model that you've generated or at least the updated computational graph, what is the method by which you would then try to introduce it into an organization or into the wild for it to be exploitable now that you know that you have a model that you can get to do whatever that specific task is that you want? Yeah. So we've actually identified quite a few different just ways to get stuff out there. One of the biggest is Hugging Face. Once DeepSeq came out, I think within a matter of hours, there were dozens, if not more, versions of Deep seek, fine tunes, quantizations,
[00:25:53] Kasimir Schulz:
things like that. Right? Quantizations is one of the big ones that we see as an attack vector because the weights and biases might be the exact same. Or not exact same, but because you quantize, but most of the structure will be the same. But there's expected changes. Right? And a lot of people don't have the ability to actually quantize on their own machines. So they rely on somebody to quantize, and they download that. And then what you might do is you might run comparisons between the original and the quantize. Quantized. However, with the computational graph, there's gonna be absolutely no difference unless you hit the trigger. So it's very different than a poison model, poison dataset, you know, fine tune like that because you might actually see small changes. But with the computational graph, because we're able to do exact changes, you don't see anything. So it would pass all benchmarks. So that's, you know, with, quantized there, with a fine tuned as well. You might see, you know, specific changes, but then most of the weights might be the same or have just very small changes. But since we're in the computational graph, we're not actually changing the weights. So there's no changes there either. And then finally, another attack surface that we identified as a big risk for just shadow logic is insider threats. With traditional code, you know, if I am an insider threat and I make a change to GitHub, it's gonna be fairly easy to read the code change because it's in plain text. However, if I'm changing a model, any change to the model is gonna be fairly large. You know, if I'm fine tuning, if I'm adding something, but it's gonna be very hard to tell what's benign and what's malicious because it's just a giant blob of binary data.
[00:27:25] Tobias Macey:
And so knowing that there is that potential for an attack vector, most people who are experimenting with models, they're going to go to a marketplace such as Hugging Face or maybe there are models that are available for consumption on things like Bedrock or Lambda Labs. And as somebody who is just saying, I just want a model. I wanna see if it does the thing that I want it to do. Okay. This passes the smell test. I'm gonna go ahead and start using it for x, y, or z. What are the ways that teams should be evaluating those models for the presence of those malicious subgraphs and ways that they can actually guard against accidental consumption of a compromised model before they ever even let it touch any of their systems?
[00:28:09] Kasimir Schulz:
I'm obligated to answer that with a small sales pitch. Not really. But hidden layer, one of the products that we sell is the ability to scan models. Whether it be hidden layer or any other one, make sure you actually scan those models because that's something that you can detect. You can detect if the computational graph is slightly different. You can detect if somebody's put a backdoor in there. So just like with traditional software, don't just download and run the code. Even if you've looked through it, there might be something that you don't know about. Use a tool. There's plenty of them out there.
[00:28:39] Tobias Macey:
In the event that you do inadvertently incorporate a compromised model into your overall system, what value do the overall approach of things like guardrails and prompt filters add to mitigate against the exploitation of that hidden subgraph?
[00:29:00] Kasimir Schulz:
Yeah. So a lot of those guardrails and filters, they won't catch a traditional backdoor because a lot of them are, you know, trained for specific inputs, specific output, detecting prop injection. Whereas, you don't have to prop inject anymore once you have a backdoor model. Right? Because you can just give it whatever input you want, whatever output you want. However, one place where those guardrails do work is now with the more systems. Instead of just detecting prompt injection coming in, you could detect are the ways the tools are being called malicious or suspicious.
This model should, in this context, only be calling fetch tool, for example. Why is it doing fetch then file system then another fetch? So keeping track of that. And we're starting to see a lot more of those kinda task tracking abilities come out. I think Microsoft actually released task tracker, which works pretty well for something like that as well.
[00:29:55] Tobias Macey:
And then the other element of exploiting a system is once you have introduced a backdoored model, you've put it out into the wild, you're waiting for somebody to consume it, detection element where for a lot of categories of malware, for somebody who is actively attacking a system, they will gain some foothold. They will place some sort of command and control software on the compromised platform so that they can then send back information or have a a two way communication path. For the case where you're waiting for somebody to consume and deploy a model, and you know that once that model is running somewhere, you have a means of exploiting it. What are the detection options for being able to understand, okay, company x, y, or z has now deployed my model. They are exposing it through this set of API endpoints. Now I can actually go and compromise it and execute whatever the actual intended attack might be.
[00:30:48] Kasimir Schulz:
Yeah. So it depends a lot on the actual model and the use case. So I think, for me, there's two main classifications of compromised models. There's the compromised models that'll actually execute arbitrary code on your system when loaded. So with those, you'll more likely to see, like, that traditional, c two frameworks and all of those. And then there's the backdoor models. And the backdoor models aren't necessarily supposed to run arbitrary code on your system, but it's just a little backdoor for you to do whatever you want. Right? So if I know that a model is going to be used to approve loans, I might have it so that I know that if I put a specific comment in, you know, anywhere in there, my loan automatically gets approved even if it's bad. I could attack it that way. Or if it's an agent, I could have it where if I have a specific command, it ignores the system prompt. That's very dangerous too. But it really depends on the attack and what the attacker can dream of. At this point, there's no limits because models are really being put anywhere and everywhere.
[00:31:50] Tobias Macey:
And in, we'll call it, traditional systems or more deterministic systems, There are obviously large categories of vulnerabilities that are out in the wild because maybe you're running an older version of the Apache or NGINX web proxy. A lot of software that is run over the Internet has various signatures that you can consume to understand, okay. This is a system that I can exploit because it's NGINX, and it's five versions behind, and it's using a version of OpenSSL that has some attack vector. So now once I've done that reconnaissance, I can then start to execute my attack, and there are massive Internet scans running constantly. So anybody who's ever run software that touches the Internet will see all kinds of logs of requests for various PHP endpoints, etcetera. I'm wondering if you're seeing similar activities and approaches for that evaluation of the presence of some of these models, whether there are standard signatures that are being exposed to say, okay. This is an LLM system. It's running this model or some ways to exfiltrate that signature of which model is running through some standard benign prompts and things like that for being able to do that detection of which model is out there and whether or not I can compromise it. Yeah. So those type of scans are a lot less for backdoor models and more for the kinda AI red teaming stuff. So we have an AI red team. There's also Garrick by NVIDIA. And one of the first steps that they do is they try to identify what actually is this model. You can identify that either by kinda how the model responds,
[00:33:24] Kasimir Schulz:
what error messages it has, because some of the different models have standard error messages or rejection messages. The models also have different control tokens. So one of the control tokens, pretty much what it is, is you have the chat templates that normally are associated with the model. So if you give it the system prompt and the user prompt, it generates the chat template that has this nice little I'm start or, you know, sys, stuff like that. And those are fairly unique across the models. So you can sometimes test, you know, how will model respond to one control token versus another control token. However, that's becoming less of a thing now because control tokens are fairly powerful for overriding system prompts. So they're actually now being kinda filtered out. But, yeah, you can do a lot of those kinda easy checks to see if this is this type of model, this this type of model. I know with this type of model, this one attack works a bit better, but we're also seeing a lot more of that identification with MLOps flame frameworks.
We actually have a few that we've exploited, and we have honey pots up, and they actually get hit fairly regularly. And then we also have been seeing it with the MCP servers. So MCP doesn't actually have authentication by default. So what we're able to do is we can scan for I'm using Shodan to look for MCP servers being hosted, and we can actually see what tools are being hosted. So we found, I think, a little over a hundred MCP servers just open on Shodan. Everything from Jira so you could read this entire company's Jira, create tickets, anything you wanted there, Gmail, Google Drive, lots of things you don't want exposed to the Internet. And keep in mind, a model doesn't even have to actually
[00:35:04] Tobias Macey:
interact with that. You could just send whatever tool you, call you want directly to it as well. Because of the fact that this landscape is moving so quickly, there are so many changes happening all of the time, and the ability for organizations to so quickly ramp up into using LLMs because of the fact that they don't require as much upfront work as a standard machine learning model development deployment setup. What are some of the elements of maturity of sort of organizational and technical maturity that are getting leapfrogged as a result that lead to a lot of these potentials for security issues in the deployment and use of these generative systems?
[00:35:49] Kasimir Schulz:
Yeah. So definitely just traditional security, is a big concern of mine. There's so many open source AI and ML projects out there. And if you look at them, almost all of them will have some sort of vulnerability. I don't think we've come across any AI ML system where that's open sourced even if it's from a company where we haven't been able to find something with a static code analyzer. So not even actually combing through and really doing vulnerability research, but just running one of our static tools against it, and we're able to exploit it. So that's definitely one of the biggest problems. The other problem that I'm seeing a lot is that companies don't necessarily fully understand what exactly they're implementing. So they're taking these foundational models, they're hooking them up to things. I've seen a lot of, you know, companies that are doing really great and innovative things, but they're rappers for, you know, Chativity with, you know, other tools being called additional information. And they just they don't know about the attack surface that these models have and the fact that they're so easy to attack. It's not like, you know, traditional security. Somebody might have to study for years to actually be able to attack a system. But with these AI systems, you could learn in a few weeks, do a CTF or two, and then you're already compromising things. For example, a while back, we reported a vulnerability for AWS.
So they had the Titan Bedrock image generator. And what they did is they added a watermarking feature to combat misinformation just because, you know, that's one of the things that was a really big issue around them. So everyone kinda rushed to get it out there. And what you were able to do was you could just change a small portion of an image, and it would so because there was, like, a replacement tool and a remove object removal part for the image generator as well, and it would generate the new image with the model mark applied. So all you ended up having to do was subtract the two images, and you had the watermark. So you could arbitrarily add a watermark to any image or remove the watermark from an image just by adding and subtracting.
So it's a lot simpler to exploit these things just because of the rapid development that's happening.
[00:38:01] Tobias Macey:
And to that point of the use of managed model providers such as OpenAI or Anthropic or AWS Bedrock, how does that change the threat vectors for organizations that are using generative models versus running it on their own hardware using something like a VLLM or some of the other model servers that they run on their own hardware?
[00:38:27] Kasimir Schulz:
Yeah. So I think the main attack surface tends to be about the same. It tends to be that people who are deploying their own models have a bit more background potentially than somebody who's just hooking up a chat g b t API. So they might be aware of certain, you know, attacks, prompt injection, both direct and indirect, control tokens, things like that. So they might be the ones who then go that next step and get the guardrail implemented or try to add in task tracker. I think with task tracker, Microsoft task tracker, you actually have to have the model locally to actually be able to implement that. So you do have more control over the security if you do it yourself. However, the models from the foundational providers are going to be stronger. So just anybody coming up is gonna have a lot harder time prompt injecting the new cloud or prompt rejecting the new chat g v t if there's a really solid system prompt than a local one. But that local one, you can add anything you want to on top of it.
[00:39:28] Tobias Macey:
And going back to the ancestry of models and the the shared logical graphs, how does the understanding of that ancestry
[00:39:47] Kasimir Schulz:
actually, love that question. So one of the things that you're able to do by understand the ancestry of the model is instead of trying to protect against all the attacks that are out there, which that's never going to work, you can either try to protect against the very specific attacks that might happen, or what you can also do is you can try to detect them. So you were talking about how do we do detection earlier. So, for example, if I'm running an image model, and I know that's the predictive classifier model, I and the attacker might try to, you know, nudge a little bit based on features in the input space.
I could do an analysis where, if an attacker is trying to do hop, skip, jump, so they're just sending a bunch of input trying to notice, you know, this little change, what's the change in the output. I can then, detect that because it's gonna be lots of very, very similar input. However, you know, the embedding space might change. So with, like, a BERT class like, distal BERT or BERT, classification model, I could see that the text is going to be almost the exact same, using, fuzzy hash algorithm. However, the actual embedding space for tokens might be different, because the semantics is different. And those are, like, the types of attacks that you can really detect if you understand the genealogy of the model.
[00:41:09] Tobias Macey:
The point about embeddings also introduces another question as far as attack vector is going back to the fact that it's not an isolated system. It is dependent on the data that it's being fed for cases such as RAG or a lot of these agentic systems where they're doing fetching and analysis and synthesis of data. In the event that you are building a RAG style system, you want to feed all of your content through an embedding model. That model itself is also potentially something that can be compromised. And I'm curious what you're seeing as some of the threats that are in the ecosystem around that portion of the life cycle of these AI systems.
[00:41:45] Kasimir Schulz:
Yeah. So, actually, so far, it's been fairly unexplored. Though I we actually have a white paper coming out in the next month or so, really exploring a bunch of new attacks about that. So I'll definitely make sure to link you that paper, but I can give a few little kinda hints and things ahead of time. So there's quite a few attacks that can happen there. For example, one of the things that you can do is that there might be certain words that look the same to a person, seem the same to a person, but the embeddings can be wildly different. Or you can have a embedding model that has a different understanding than the underlying LLM. So for example, with a prompt injection detector or spam detector, it might use one set of embeddings for the guardrail model, but another set for the underlying model. So what happens if you drive the embeddings apart? You know, you can really start doing a lot of interesting attacks that way. No. It's
[00:42:42] Tobias Macey:
definitely an interesting aspect of which just makes it so much more challenging to do the analysis and evaluation of what models do I use, how do I build my system, what is the architecture, which probably leaves a lot of people to just throw up their hands and say, I'll just throw money at somebody who I think is doing it the right way.
[00:43:00] Kasimir Schulz:
Yeah. Exactly.
[00:43:03] Tobias Macey:
And in your experience of working in this space, in particular, the work you've been doing around the shadow logic and shadow genes analysis, what are some of the most interesting or innovative or unexpected ways that you've seen the knowledge of model ancestry applied both for offensive and defensive uses?
[00:43:23] Kasimir Schulz:
Yeah. So I think for offensive, it was definitely being able to attack the camera the way that we did. Just being able to generate everything in a nice easy way and then just having 20% of it go over. That was awesome. On the defensive side, though, definitely what I was talking about with you can fine tune your defenses for it. Yeah. I would say that's, you know, the the main thing there is just being able to fine tune any defenses that you want for a specific model. And in your work
[00:43:52] Tobias Macey:
of exploring the space of AI and ML security, what are some of the most interesting or unexpected or challenging lessons that you've learned? I learned that,
[00:44:04] Kasimir Schulz:
I do not understand AI and ML, at all, which has been so much fun. And, honestly, I think it's helped me out a lot because all of the we have so many brilliant data scientists, and they build all these awesome models. And because I don't really understand them necessarily all the time, I'll just poke them the wrong way, and it'll all fall apart. And then I get to learn something completely new. So it's been really, really fun experience. But I think a lot of people have some misunderstandings for those models. And I think that's definitely an experience that I want a lot of other people to have too, especially our data scientists too. Just because sometimes I'll poke something in the wrong way. I won't understand why the output happened that way. I'll go to the data scientist. They'll be like, oh, we don't know either. And then we actually look at it, and we understand some fundamental piece of the model that actually has a huge security impact that we never even considered before just because so many of us treat these AI models as just black boxes and don't try to understand them. Where in reality, they are black boxes and hard to understand, but not to the level that people expect.
[00:45:15] Tobias Macey:
And for teams and individuals and organizations that are investing in the space of ML and AI systems, which is most of them at this point, What are some of the ways that you suggest they keep up to date with the wide and broadening set of attack vectors and threats that are in the ecosystem and means to mitigate and guard against them?
[00:45:41] Kasimir Schulz:
That's always a tough one. That's also a tough one for traditional security. There's going to be certain companies that publish certain work. So we always any attacks that we find, once we actually go through and report them, we publish them. So there's always certain research or blogs that you can go to to read that. On top of that, just staying on top of who are the people who tend to repost the news for you. You might not have to go and find everything, but there might be somebody on x or Mastodon or Blue Sky who will actually go through and post all the news feeds. Let them do the work, and then you can kinda look through.
But it's definitely what I recommend is find the data sources that you are willing to keep up with and try to get as many of those data sources being ones that take in other data sources and kind of filter it down for you. You're going to miss stuff. You're always gonna miss stuff, but that gets you the most important stuff, especially if you have enough data sources. Because once you see, you know, 10 of the people all posted the same exact link, that's probably something I should read rather than just one of them posting something.
[00:46:48] Tobias Macey:
Yeah. One of the newsletters that I've found useful for infosec as well well as AI is the TLDR family of newsletters, and I can add a link in the show notes for that.
[00:46:58] Kasimir Schulz:
Yeah. It's a it's a great one. I'm actually subscribed to that as well. It's part of the reason why I did the MCP blog because I started seeing it a little bit more, and then, all of a sudden, I saw that 10 of the links, not even the AI portion were all titled MCP something. So I was like, okay. I need to actually really look into this.
[00:47:17] Tobias Macey:
Absolutely. Are there any other aspects of the work that you've been doing on shadow logic and shadow genes and the ways that that can and should inform the selection and application of ML models that we didn't discuss yet that you'd like to cover before we close out the show? I mean, the main thing I wanna talk about there is just make sure you scan your models and make sure you're aware of what you're actually using.
[00:47:41] Kasimir Schulz:
Be that shadow genes, be that anything else, but that's just something that's really important. Don't just blindly download and run.
[00:47:50] Tobias Macey:
Absolutely. Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you're doing and the rest of Hidden Layer, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gaps in the tooling technology or human training that are available for AI systems today.
[00:48:08] Kasimir Schulz:
Yes. I think the biggest gap is definitely the learning portion of it. I think we need to do a better job at informing the developers who are actually gonna be implementing it everywhere. Just what are the attacks that are out there? There's already an issue of that with a lot of, developers don't have security experience. I think when I went to I did my undergrad, and I only had one class that was in security before I got into grad school even though I did computer science. So it's it's not hard to attack these systems. So it shouldn't be hard to give, you know, an hour class here or there, make something free available so people can hack away at it, and just learn about what you need to actually look out for. Because just having even if you don't understand it, even if you can't fully do it, just knowing that it's there is going to make you a lot better at securing it.
[00:49:06] Tobias Macey:
And, I guess, on that point too, what is your opinion on the potential for these AI systems to also act in the guidance portion of development and deployment to say, hey. You really shouldn't do that there, or don't forget to check on this thing over there.
[00:49:23] Kasimir Schulz:
I am the first to say that I think AI has sped me up a lot in my day to day work, but you also have to be very careful about how you use it. So a lot of people are using AI to do all of the coding for them, and that's going to start causing a lot of problems down the road, I think. The way that I use it is I use it if I have to do rapid prototyping. If I understand how something works, for example, I think maybe once a month, I'll ask it, write a Python snippet to find all the files recursively of this file type in their directory. It's stuff that I used to have to stack overflow and then modify, but I understand the core concept of how it works. So what it lets me do is because I understand it, it's something I can easily test.
I'm not wasting time trying to debug the code that it made, not wasting time, you know, potentially debugging an issue down the road. So it's really, really amazing for those simple one off things, the prototyping, the looking up documentation, anything that you can very quickly verify is working correctly.
[00:50:28] Tobias Macey:
Absolutely. Yeah. Trust but verify.
[00:50:31] Kasimir Schulz:
Exactly.
[00:50:32] Tobias Macey:
Alright. Well, thank you very much for taking the time today to join me and share the work that you and your team are doing on the Shadow Logic and Shadow Genes work and just giving us an overview of the threat landscape for these AI systems. It's a definitely very important aspect of the work that is being done and something that everybody should definitely be paying a lot of attention to and probably more than they already do. So thank you for all the time and energy that you're putting into helping to surface some of these attack vectors and ways to mitigate them. And I hope you enjoy the rest of your day. Thank you too. And thank you so much for having me on the podcast today.
[00:51:14] Tobias Macey:
Thank you for listening. And don't forget to check out our other shows, the Data Engineering Podcast, which covers the latest in modern data management, and podcast dot in it, which covers the Python language, its community, and the innovative ways it is being used. You can visit the site at the machinelearningpodcast.com to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hoststhemachinelearningpodcast dot com with your story. To help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.
Hello, and welcome to the AI Engineering podcast, your guide to the fast moving world of building scalable and maintainable AI systems. Your host is Tobias Macy, and today, I'm interviewing Casimir Schultz about the relationships between the various models on the market and how that information how that information helps with selecting and protecting models for your applications. So, Kasimir, can you start by introducing yourself? Yeah. Hi. My name is Kasimir Schultz. I'm the director of security research at Hidden Layer. And do you remember how you first got started working in ML and AI? It's actually been a interesting journey. So I started with a more traditional security background. And, actually, even before that, more in the software engineering side of things. And,
[00:00:49] Kasimir Schulz:
I was doing traditional security zero day research to try and find vulnerabilities and just anything out there. And I came across Hidden Layer, and I actually initially joined to do security in just anything AI related. So the AI frameworks, AI model formats, and things like that. And, once I joined Hidden Layer, I kinda went down the rabbit hole machine learning, and I've been learning from our data scientists, learning from just breaking things and reverse engineering things. So it's been a fun journey there. Yeah. It's definitely a very important area of research even before the current epoch of all of the generative AI systems just because of the fact that there is so much potential for issues
[00:01:28] Tobias Macey:
and attacks because of the probabilistic nature of the software where you can't just run a static analyzer and say, oh, that's where the problem is. That's what I need to patch. You have to worry about so much of the overall process that goes into building these models and then operating them and maintaining them because it's not just a single application or a single process that you need to worry about. It's all of the data that goes into it, all of the code that goes into building it, and all of the process around actually keeping it secure in in its actual runtime.
[00:01:58] Kasimir Schulz:
And, I mean, I'm seeing that a lot more now as well. So one of the biggest issues I'm seeing nowadays is just all of this agentic stuff. People are just trying to innovate, which is awesome. There's so much cool stuff around that. But the issue with that is people aren't really considering what goes into it and what does the model have access to. So people aren't considering what the model actually has access to and then the frameworks that it can interact with. For example, I was just working and published a blog on MCP. So the model context protocol last week. It's just been a hot topic. Everyone was talking about it. I was at a conference last week and I had great discussions with people about it as well. And one of the things that we noticed is that a lot of traditional security vulnerabilities are in these new formats and these new agentic systems. And then a lot of these just agentic systems also have these innate systematic vulnerabilities and weaknesses as well. So for MCP, one of the things that we discovered is that a lot of the MCP servers that were written and that were out there and recommended by people, they allowed code execution.
And the reason that they allowed code execution is because they were running on your system. Right? So if you're connected to cloud, you're working through it, everything's fine, nothing bad should happen. And people are writing it along the context of that's the only attack plane that there actually is. However, once you start integrating more MCP servers, you have a lot more attack surface because, for example, reading a file on your local file system. What do you do if there's an indirect prompt injection in there that then tells you to run, arbitrary code with another MCP server? Or for example, we were using the Ghidra and binary ninja MCP server to reverse engineer binary.
And in that malicious binary, so the malware, there was a prompt injection in there, and it started executing arbitrary code on our system as well. And even if you don't have arbitrary code execution, just kind of the mesh of all of these different tools that agents have nowadays allows for just a whole new area of attack surface. So one of the things that MCP does is and MCP cloud, desktop is you have all the permission management. So you can allow a tool to run once or you can allow it to run for the session. And then every new tool, you have to give permissions. However, once you're actually running through a tool, sometimes one single prompt from you requires the same tool be to be called, you know, fifteen, twenty times. So you're gonna click allow for the session. So what we're able to do was just by asking Claude to download and summarize a document for me, because it downloaded the document, it used fetch. And then because it downloaded and read the document, it used the file system tools. And then at that point, there was a prompt injection in there that actually let us read all of the file system and then exfiltrate the data since we already have those tool permissions.
[00:04:39] Tobias Macey:
Yeah. That's definitely it's definitely a combinatorial explosion of attack surface by virtue of all of these integrations that you're trying to put together where, up until now, integration was a task in and of itself, and so it required a lot more care and intention behind it. Whereas now you're handing a lot of that over to these black box systems that you don't have full visibility into what they're doing or what they might be doing. And it's all in the name of convenience, and that that's always the challenge with security is how much convenience to allow for and how much to lock things down. So the more secure it is, the harder it is to use oftentimes unless the security was baked in from the beginning, which it's sounding like was not really the case with these MCP servers.
[00:05:25] Kasimir Schulz:
Yeah. Exactly. But it's a old cat and mouse game that's been around for a long time in security, so we'll get there. Absolutely. So broadening the scope a little bit more, I'm wondering if you can give some overview about the current
[00:05:39] Tobias Macey:
state of the threat landscape for ML and AI systems beyond just this model context protocol that has taken the attention for the past couple of weeks?
[00:05:48] Kasimir Schulz:
Yeah. Of course. So when I first got into the field, I told you I came from a traditional security background. So one of my first task was kinda flushing out all of the low hanging fruit. And while the field definitely has done a much better job at getting more secure in a traditional security standpoint, for example, I think it was maybe a month or two ago. Time's been flying like crazy. But, PyTorch now does Torch load safe by default. So for over a year and a half that I've been working at hidden layer, that was one of the things that we kept seeing. There were over 500,000 projects on GitHub that had potential code execution just because they were loading a machine learning model. And we're seeing a lot of that not just with, necessarily the models themselves.
At this point, I feel like most people know a lot of these models have pickle inside and know to kinda use them safely, but we're seeing these with the safe formats as well. So everyone assumes safe tensors is safe. However, just because the file itself doesn't execute code, doesn't mean that just by loading it, itself doesn't execute code doesn't mean that just by loading it, you might not execute code. So with DeepSeek and all of these new architectures that are coming out, you have to have arbitrary code or trust remote code on when you load them through Hugging Face transformers. And what a lot of people will do is they'll look through the code. The code will look fine, especially because a lot of times what happens is that they copy the code directly from the transformers repository, just make whatever changes they have to. However, by using transformers to load models, you can still have arbitrary code executed that's not actually in the repository that you're looking at. That's this repository side loading that we're seeing a lot more. There's actually a few instances of it out on Hugging Face. And pretty much what happens there is if you have trust remote code on for Hugging Face, transformers and you're loading in a safe tensors file, it'll look at the config dot JSON. And then that config dot JSON, it'll map what classes are stored in what files. However, it can also point at other repositories.
So you might be thinking that the code that you're running is the deep secret repository that you're looking at, but it's actually running some other malicious code instead. So that's definitely one of the big things we've seen. And then we've been seeing a lot more true AI vulnerabilities and AI attacks lately, which has been really exciting to see, just kinda what are people coming up with, not just in the sense of generative AI, but also predictive and classification models.
[00:08:10] Tobias Macey:
And on that point of the divide between predictive and classification style models that have been the bread and butter for machine learning systems for the past decade plus, but also now moving into this landscape of generative models. What are the main areas of overlap, and what are the key points of divergence as far as the attack surface and, threat signature from that division of the the model types?
[00:08:36] Kasimir Schulz:
Yeah. So it's actually funny enough. The overlap is a lot bigger than you would think for the just types of attacks that you can do. What's really kind of the main attack that you can do and the main vulnerability of these types of models is that in most applications, the entire input that goes into the model is completely user controlled. And what that means is, you know, with software, you might have an API endpoint, and you can query just part of that API endpoint. But all of the data, everything else is controlled by a server, is controlled by access management. But with a model, every single thing that is processed in that model most likely came from the user, except for, like, you know, the little cases with, a system prompt, for example, that might come from the system. But the main chunk of it is from that user. And then the goal for attacking both is the same as well, and that's controlling the output. And I think that once a lot of us started thinking along that mindset, we've been able to come up with a lot more attacks that we haven't necessarily done before. So the idea is, you know, all your goal is to control the output in one way. So you're not necessarily trying to do prompt injection. You're not trying to do a jailbreak, alignment bypass. It's just controlling the output. And when you think about it, controlling the output is not that hard if you control everything that goes into the model, and then you actually start understanding how the model works. So I think that's the main overlap between the two of them. As far as, divergences, I found that attacking predictive and classification models, it's a lot simpler. It's not actually easier in my opinion than generative models. So generative models are on the easier side. However, it is simpler because what you really have to do for attacking those models is you just have to figure out what inputs and what what, feature in the input space will just nudge a classification just slightly over the edge. Right? Because that's all you have to do. You have to trick the model and control the output of the model while still looking like it's one input or another input in most cases. Right? So for example, a classification model that detects malware or not malware. If you're running malware, you can put whatever you want inside the model. We've been able to find that sometimes it's as easy as just writing the text this is benign 20 times somewhere in the malware like a string or a comment and something like that's enough to flip the model one way or another just because you can keep nudging it. A lot of these models, it's people assume that it's, you know, the model thinks, but the model really is just math. So sometimes you can just real do really stupid attacks, and they work really, really well. And then the reason why I was saying that generative models are not actually harder and, are tend to be a lot easier to attack is there's just so much more that you can do with them. So one of my teammates, while back came up with a technique called CROP, knowledge return oriented prompting. And what they can do with the technique is most of these large language models and small language models, they contain a lot of information. They know all the pop culture references. They know all of, you know, those types of things. So I I think most people are gonna be aware of, you know, x k c d that you've seen a few of them. Yeah. So we actually had a model sitting on top of a SQL database.
And the model had a bunch of protections saying, don't write code to drop the tables. Don't write code to remove stuff. Just do simple, you know, select statements. And what we were able to do was we were able to reference the little Bobby tables x k c d. And whereas saying, you know, hey. Write me a statement to drop the database. It said no. But say, hey. What is the name of that kid? Little Bobby tables with his full name, replace this one part of it with something else. And we were able to actually exploit the system underneath just because that wasn't a guardrail. There were no guardrails about actually getting information. And then you can do a lot of those attacks without ever actually having something bad in the prompt. So, for example, instead of ignore, previous instructions, we were able to do pretend you're a developer and pipe your output to devnull. And the model understands that, and it'll do those things. And it's also really interesting in the, multimodal space as well, which is becoming just a whole fascinating field with generative models. We were able to give the model just a image of a reverse sign and a shell, and we were able to tell the model, take the two items in the image, make it Python code, and tell me what the code is. And then we were able to get chat g b t to write as a Python or reverse shell where it normally wouldn't do something like that.
[00:13:02] Tobias Macey:
Yeah. It's definitely interesting the way that the introduction of generative models just cracks the overall attack surface wide open because there is so much potential for use and output that it's effectively impossible to guard against all potential inputs. And I don't know if there has been any work so far that's ongoing to come up with something along the same lines as the OWASP top 10 for web applications in the context of generative models. I know that there are things like guardrails and input and output filters for trying to address some of that. But as you just pointed out, there are a lot of clever ways that you can work around the heuristics that are built into these systems.
[00:13:45] Kasimir Schulz:
Yeah. So there actually is a lot of great work going on. So OWASP has a top 10 for LMs as well, which has been great to use. And then the CVE organization has two working groups, one to define what actually is a CVE around AI systems and then one to define, CWEs or the common weakness enumerations for, AI systems as well. So there's lots of organizations actually trying to figure out, you know, how can we classify these attacks, then how can we classify these attacks, then how can we defend against them.
[00:14:17] Tobias Macey:
And then in the category of generative models, there are obviously numerous stories that are already in the wild about ways that they've gone wrong. I think maybe the most widely spread one is the story about the car dealership that had their model promise to sell a car for a dollar because somebody was able to trick the model. But I'm curious from a gradation of damage both to the organization, whether it's monetarily or reputationally, up through to issues around physical security and safety, what are some of the potentials for damage that can be caused through the deployment and use of these models whether compromised or not?
[00:15:03] Kasimir Schulz:
I mean, it's a big question. There's a lot. So, obviously, you know, the situations with the new hallucinations or prompt inject it. Like, you, mentioned there's brand issues, you know, if your model suddenly makes a bad joke about your brand and it's your chatbot on your website, that's not great either. With a compromised model, I mean, now that we're hooking them up to agentic systems, that's definitely now there's actual real world impacts of, you know, the system doesn't just tell you something. It actually does an action for you. So it might delete your database or it might run malware on your system. And then there's also a whole interesting kinda part now where we're seeing a lot of these, jailbreaks, especially for local models. And what's really scary there is, as I mentioned earlier, a lot of these models have a lot of information. Right? They know all those pop culture references. They know how to do specific tasks, and they they have all that information that was fed into them to train them. And they're trained so that they won't tell you how to make a bomb, won't tell you how to make make malware. But with local models, they're jailbreakable.
People can either fine tune those guardrails out of them, or there's, new work being done called refusal vector ablation. It's a really cool paper a while back where they realized that the refusal direction for a model was pretty much mediated by one single vector in the model. And what you could do is you could just ablate that, vector. So I actually have five three, five four, llama three, and gemma all fully jailbroken on a local machine, done on a CPU. So I was able to, you know, completely remove those guardrails without a fancy GPU, actually, on the computer I'm speaking to you on today. But what's kind of scary about those things is that even five three where it was trained on synthetic data, it's still able to answer how to make a bomb, how to make math, things like that. And now, because it's on my local system, those searches are completely untraceable. It's not like, you know, asking chat g b t how to make a bomb like the person who did with the cyber truck explosion a few months ago in Vegas. It's not like Google search for somebody can monitor that. So this is all just completely unmonitored on every anyone's laptop now.
[00:17:14] Tobias Macey:
Yeah. That's definitely a scary scenario. The availability of information by virtue of it having been compressed into these models, to your point, without any sort of network requirement to actually go out and fetch that information. Yeah. The other interesting shift in terms of the AI landscape that was brought in by the rise of generative models is the widespread use now of foundation models as a building block for systems that are being put into production and the fact that most of those foundation models are originating from a small handful of organizations and companies.
And I'm wondering what you see as the level of risk both in terms of platform risk from a organizational perspective, but more specifically, security risk of these foundation models being distributed through systems such as Hugging Face and the fact that so many organizations are relying on a small pool of models to then go and build a wide breadth of services on top of?
[00:18:16] Kasimir Schulz:
Yeah. So a lot of smaller companies don't have the ability to create those foundational models. So foundational models definitely are, you know, something that's useful, something that's good for just the majority of people. However, because a lot of them are so similar, a lot of the same techniques work throughout all of them. But because they're managed by different organizations, we tend to have to report pretty much the exact same issue to every single model provider. And the reason for that is they don't talk to each other as much just because I mean, why would you in such a situation? Right? And we're actually gonna be publishing some research next week, where we have a single prompt that works on every single foundational model, and can be used to either jailbreak or leak the system prompt. So definitely is a bigger supply chain issue than people might think about.
[00:19:04] Tobias Macey:
On that note of the commonalities between these models, you recently published some work that you've been doing at Hidden Layer around the shadow gene where you're figuring out what are the common subgraphs within these model structures and these neural networks that are shared and reused across different families of models and probably even across families just because it's such a a common pattern in deep learning and neural network design. I'm wondering if you can talk to some of the ways that that information about the shared subgraphs informs the potentials for bad actors to identify models to target for these types of attacks that are known to work against one particular logic path.
[00:19:47] Kasimir Schulz:
Yeah. Before I go into ShadowJeans, though, I do wanna talk a bit about Shadow Logic, which is what we published right before and which is actually what led us to ShadowJeans. So I think last June ish, I had started working on a lot of these new backdoors that were coming out. So the backdoors were poison datasets, backdoors that were fine tuned into models. And one of the biggest issues I saw was that if you fine tuned one of these models again, a lot of times the backdoor would disappear just because they were put in there through the fine tune. And I wanted something that would persist across the fine tunes, persist across model conversions. And the reason for that is because a lot of times, as you said, somebody might take a model off of Hugging Face, but then they might adjust it slightly for their use case. And that's when I actually looked into it a bit more and found out about the computational graphs. So the computational graphs are I like saying that the weights and biases are almost like the memories of the model, and then the computational graph is like the synapses in the brain. So it's actually how the input traverses the model, how each operation gets applied, how the weights and biases get applied, and then finally, we get to the output. And normally, these are just a lot of mathematical operations as well as, you know, just the more AI mathematical operations. So like convolutions and sigmoids and stuff like that. But what we were able to do was because it really is just a pathway of mathematical operations.
To me, it felt a lot like traditional byte code, an assembly code. And I was able to start coding in any actual logic I wanted. So I was able to code in logic for a Resnick, image classifier where if a pure red pixel, so two fifty five zero zero, was present anywhere in the photo, it would change the class from one class to another. I was able to do it with a YOLO model. So a lot of YOLO models are in, security cameras. We actually hacked a security camera a while back that actually contained the YOLO model in it. And by lifting up a cup, so if a cup and a person were present in the same time, the person would disappear. So you could walk up to the camera and you'd be completely invisible, but only if you held a cup or only if you had a cup and glasses on. We were able to do it with generative models as well. So with five three, have it where the model would respond completely normally except when you knew a specific passphrase. And then you could give the passphrase in whatever output you wanted, and it would just output the actual model that you wanted. But while looking through all of these, we were spending days way too much time looking at the computational graphs through Metron, which is a great open source viewer for these graphs.
And looking through them, we started realizing that we were recognizing what the actual models were and what families they were coming from. So at the start, we had to, you know, see, oh, what model is this? Where is something? And by the end, one of us sent one of the other people a new model. We could immediately tell, oh, that's a BERT model or that's a resident model. And, even if it was a model that we'd never seen before, like, if we had never seen Roberta before, we were able to tell, oh, that's where the transformers is. That's what I need to change there. And that's what led us to the idea of shadow genes. So what we realized was we were recognizing these repeated subgraphs inside of the models because they were the same throughout all these other models. So even if the model had slight modifications, if you wanted one specific flare to it, you still have this core components.
And what we were able to do with ShadowJeans is we were able to write signatures. So the signatures match a specific subgraph inside the greater graph. And with these signatures, we can tell if a model is a specific family, what components are in the model, and then even if the model was derived from another model. So going back to the Roberta example, if we had never seen Roberta before, we had never written a signature for Roberta. We have seen BERT before. So we would have been able to tell you that the Roberta model was derived from the BERT model even if we'd never seen it before. And what's really cool about that, the attack space, which is what you're asking about, is that a lot of these models have certain attacks, not vulnerabilities, but attacks that kinda transfer between the models.
So, for example, the security camera that we hacked a while back, it was, one of the Wyze security cameras, which are these budget home cameras, and they have the AI model actually on the device now. But going through the model, we could see that they had a fine tuned version of YOLO there. It was their own model format too, so it was all wonky, and it was quantized as well. But because we could tell it was a YOLO model, we were able to generate a bunch of attacks on our computers with a standard YOLO model, and about 20 of the successful attacks transferred to the security camera. So we were able to train on something where we actually had the ability to scale up GPUs instead of having to send one image through the camera, which took maybe ten seconds to process each time.
[00:24:50] Tobias Macey:
Bringing us back around to the supply chain vectors, Once you have a model, you're aware of the different subgraphs. You're able to add in some of these alternate logic paths because of the fact that you know that, oh, hey. This is doing this mathematical computation. And if I want it to go this other way in this subset of use cases, I can add in an additional graph to the neural network. And then once you have that model that you've generated or at least the updated computational graph, what is the method by which you would then try to introduce it into an organization or into the wild for it to be exploitable now that you know that you have a model that you can get to do whatever that specific task is that you want? Yeah. So we've actually identified quite a few different just ways to get stuff out there. One of the biggest is Hugging Face. Once DeepSeq came out, I think within a matter of hours, there were dozens, if not more, versions of Deep seek, fine tunes, quantizations,
[00:25:53] Kasimir Schulz:
things like that. Right? Quantizations is one of the big ones that we see as an attack vector because the weights and biases might be the exact same. Or not exact same, but because you quantize, but most of the structure will be the same. But there's expected changes. Right? And a lot of people don't have the ability to actually quantize on their own machines. So they rely on somebody to quantize, and they download that. And then what you might do is you might run comparisons between the original and the quantize. Quantized. However, with the computational graph, there's gonna be absolutely no difference unless you hit the trigger. So it's very different than a poison model, poison dataset, you know, fine tune like that because you might actually see small changes. But with the computational graph, because we're able to do exact changes, you don't see anything. So it would pass all benchmarks. So that's, you know, with, quantized there, with a fine tuned as well. You might see, you know, specific changes, but then most of the weights might be the same or have just very small changes. But since we're in the computational graph, we're not actually changing the weights. So there's no changes there either. And then finally, another attack surface that we identified as a big risk for just shadow logic is insider threats. With traditional code, you know, if I am an insider threat and I make a change to GitHub, it's gonna be fairly easy to read the code change because it's in plain text. However, if I'm changing a model, any change to the model is gonna be fairly large. You know, if I'm fine tuning, if I'm adding something, but it's gonna be very hard to tell what's benign and what's malicious because it's just a giant blob of binary data.
[00:27:25] Tobias Macey:
And so knowing that there is that potential for an attack vector, most people who are experimenting with models, they're going to go to a marketplace such as Hugging Face or maybe there are models that are available for consumption on things like Bedrock or Lambda Labs. And as somebody who is just saying, I just want a model. I wanna see if it does the thing that I want it to do. Okay. This passes the smell test. I'm gonna go ahead and start using it for x, y, or z. What are the ways that teams should be evaluating those models for the presence of those malicious subgraphs and ways that they can actually guard against accidental consumption of a compromised model before they ever even let it touch any of their systems?
[00:28:09] Kasimir Schulz:
I'm obligated to answer that with a small sales pitch. Not really. But hidden layer, one of the products that we sell is the ability to scan models. Whether it be hidden layer or any other one, make sure you actually scan those models because that's something that you can detect. You can detect if the computational graph is slightly different. You can detect if somebody's put a backdoor in there. So just like with traditional software, don't just download and run the code. Even if you've looked through it, there might be something that you don't know about. Use a tool. There's plenty of them out there.
[00:28:39] Tobias Macey:
In the event that you do inadvertently incorporate a compromised model into your overall system, what value do the overall approach of things like guardrails and prompt filters add to mitigate against the exploitation of that hidden subgraph?
[00:29:00] Kasimir Schulz:
Yeah. So a lot of those guardrails and filters, they won't catch a traditional backdoor because a lot of them are, you know, trained for specific inputs, specific output, detecting prop injection. Whereas, you don't have to prop inject anymore once you have a backdoor model. Right? Because you can just give it whatever input you want, whatever output you want. However, one place where those guardrails do work is now with the more systems. Instead of just detecting prompt injection coming in, you could detect are the ways the tools are being called malicious or suspicious.
This model should, in this context, only be calling fetch tool, for example. Why is it doing fetch then file system then another fetch? So keeping track of that. And we're starting to see a lot more of those kinda task tracking abilities come out. I think Microsoft actually released task tracker, which works pretty well for something like that as well.
[00:29:55] Tobias Macey:
And then the other element of exploiting a system is once you have introduced a backdoored model, you've put it out into the wild, you're waiting for somebody to consume it, detection element where for a lot of categories of malware, for somebody who is actively attacking a system, they will gain some foothold. They will place some sort of command and control software on the compromised platform so that they can then send back information or have a a two way communication path. For the case where you're waiting for somebody to consume and deploy a model, and you know that once that model is running somewhere, you have a means of exploiting it. What are the detection options for being able to understand, okay, company x, y, or z has now deployed my model. They are exposing it through this set of API endpoints. Now I can actually go and compromise it and execute whatever the actual intended attack might be.
[00:30:48] Kasimir Schulz:
Yeah. So it depends a lot on the actual model and the use case. So I think, for me, there's two main classifications of compromised models. There's the compromised models that'll actually execute arbitrary code on your system when loaded. So with those, you'll more likely to see, like, that traditional, c two frameworks and all of those. And then there's the backdoor models. And the backdoor models aren't necessarily supposed to run arbitrary code on your system, but it's just a little backdoor for you to do whatever you want. Right? So if I know that a model is going to be used to approve loans, I might have it so that I know that if I put a specific comment in, you know, anywhere in there, my loan automatically gets approved even if it's bad. I could attack it that way. Or if it's an agent, I could have it where if I have a specific command, it ignores the system prompt. That's very dangerous too. But it really depends on the attack and what the attacker can dream of. At this point, there's no limits because models are really being put anywhere and everywhere.
[00:31:50] Tobias Macey:
And in, we'll call it, traditional systems or more deterministic systems, There are obviously large categories of vulnerabilities that are out in the wild because maybe you're running an older version of the Apache or NGINX web proxy. A lot of software that is run over the Internet has various signatures that you can consume to understand, okay. This is a system that I can exploit because it's NGINX, and it's five versions behind, and it's using a version of OpenSSL that has some attack vector. So now once I've done that reconnaissance, I can then start to execute my attack, and there are massive Internet scans running constantly. So anybody who's ever run software that touches the Internet will see all kinds of logs of requests for various PHP endpoints, etcetera. I'm wondering if you're seeing similar activities and approaches for that evaluation of the presence of some of these models, whether there are standard signatures that are being exposed to say, okay. This is an LLM system. It's running this model or some ways to exfiltrate that signature of which model is running through some standard benign prompts and things like that for being able to do that detection of which model is out there and whether or not I can compromise it. Yeah. So those type of scans are a lot less for backdoor models and more for the kinda AI red teaming stuff. So we have an AI red team. There's also Garrick by NVIDIA. And one of the first steps that they do is they try to identify what actually is this model. You can identify that either by kinda how the model responds,
[00:33:24] Kasimir Schulz:
what error messages it has, because some of the different models have standard error messages or rejection messages. The models also have different control tokens. So one of the control tokens, pretty much what it is, is you have the chat templates that normally are associated with the model. So if you give it the system prompt and the user prompt, it generates the chat template that has this nice little I'm start or, you know, sys, stuff like that. And those are fairly unique across the models. So you can sometimes test, you know, how will model respond to one control token versus another control token. However, that's becoming less of a thing now because control tokens are fairly powerful for overriding system prompts. So they're actually now being kinda filtered out. But, yeah, you can do a lot of those kinda easy checks to see if this is this type of model, this this type of model. I know with this type of model, this one attack works a bit better, but we're also seeing a lot more of that identification with MLOps flame frameworks.
We actually have a few that we've exploited, and we have honey pots up, and they actually get hit fairly regularly. And then we also have been seeing it with the MCP servers. So MCP doesn't actually have authentication by default. So what we're able to do is we can scan for I'm using Shodan to look for MCP servers being hosted, and we can actually see what tools are being hosted. So we found, I think, a little over a hundred MCP servers just open on Shodan. Everything from Jira so you could read this entire company's Jira, create tickets, anything you wanted there, Gmail, Google Drive, lots of things you don't want exposed to the Internet. And keep in mind, a model doesn't even have to actually
[00:35:04] Tobias Macey:
interact with that. You could just send whatever tool you, call you want directly to it as well. Because of the fact that this landscape is moving so quickly, there are so many changes happening all of the time, and the ability for organizations to so quickly ramp up into using LLMs because of the fact that they don't require as much upfront work as a standard machine learning model development deployment setup. What are some of the elements of maturity of sort of organizational and technical maturity that are getting leapfrogged as a result that lead to a lot of these potentials for security issues in the deployment and use of these generative systems?
[00:35:49] Kasimir Schulz:
Yeah. So definitely just traditional security, is a big concern of mine. There's so many open source AI and ML projects out there. And if you look at them, almost all of them will have some sort of vulnerability. I don't think we've come across any AI ML system where that's open sourced even if it's from a company where we haven't been able to find something with a static code analyzer. So not even actually combing through and really doing vulnerability research, but just running one of our static tools against it, and we're able to exploit it. So that's definitely one of the biggest problems. The other problem that I'm seeing a lot is that companies don't necessarily fully understand what exactly they're implementing. So they're taking these foundational models, they're hooking them up to things. I've seen a lot of, you know, companies that are doing really great and innovative things, but they're rappers for, you know, Chativity with, you know, other tools being called additional information. And they just they don't know about the attack surface that these models have and the fact that they're so easy to attack. It's not like, you know, traditional security. Somebody might have to study for years to actually be able to attack a system. But with these AI systems, you could learn in a few weeks, do a CTF or two, and then you're already compromising things. For example, a while back, we reported a vulnerability for AWS.
So they had the Titan Bedrock image generator. And what they did is they added a watermarking feature to combat misinformation just because, you know, that's one of the things that was a really big issue around them. So everyone kinda rushed to get it out there. And what you were able to do was you could just change a small portion of an image, and it would so because there was, like, a replacement tool and a remove object removal part for the image generator as well, and it would generate the new image with the model mark applied. So all you ended up having to do was subtract the two images, and you had the watermark. So you could arbitrarily add a watermark to any image or remove the watermark from an image just by adding and subtracting.
So it's a lot simpler to exploit these things just because of the rapid development that's happening.
[00:38:01] Tobias Macey:
And to that point of the use of managed model providers such as OpenAI or Anthropic or AWS Bedrock, how does that change the threat vectors for organizations that are using generative models versus running it on their own hardware using something like a VLLM or some of the other model servers that they run on their own hardware?
[00:38:27] Kasimir Schulz:
Yeah. So I think the main attack surface tends to be about the same. It tends to be that people who are deploying their own models have a bit more background potentially than somebody who's just hooking up a chat g b t API. So they might be aware of certain, you know, attacks, prompt injection, both direct and indirect, control tokens, things like that. So they might be the ones who then go that next step and get the guardrail implemented or try to add in task tracker. I think with task tracker, Microsoft task tracker, you actually have to have the model locally to actually be able to implement that. So you do have more control over the security if you do it yourself. However, the models from the foundational providers are going to be stronger. So just anybody coming up is gonna have a lot harder time prompt injecting the new cloud or prompt rejecting the new chat g v t if there's a really solid system prompt than a local one. But that local one, you can add anything you want to on top of it.
[00:39:28] Tobias Macey:
And going back to the ancestry of models and the the shared logical graphs, how does the understanding of that ancestry
[00:39:47] Kasimir Schulz:
actually, love that question. So one of the things that you're able to do by understand the ancestry of the model is instead of trying to protect against all the attacks that are out there, which that's never going to work, you can either try to protect against the very specific attacks that might happen, or what you can also do is you can try to detect them. So you were talking about how do we do detection earlier. So, for example, if I'm running an image model, and I know that's the predictive classifier model, I and the attacker might try to, you know, nudge a little bit based on features in the input space.
I could do an analysis where, if an attacker is trying to do hop, skip, jump, so they're just sending a bunch of input trying to notice, you know, this little change, what's the change in the output. I can then, detect that because it's gonna be lots of very, very similar input. However, you know, the embedding space might change. So with, like, a BERT class like, distal BERT or BERT, classification model, I could see that the text is going to be almost the exact same, using, fuzzy hash algorithm. However, the actual embedding space for tokens might be different, because the semantics is different. And those are, like, the types of attacks that you can really detect if you understand the genealogy of the model.
[00:41:09] Tobias Macey:
The point about embeddings also introduces another question as far as attack vector is going back to the fact that it's not an isolated system. It is dependent on the data that it's being fed for cases such as RAG or a lot of these agentic systems where they're doing fetching and analysis and synthesis of data. In the event that you are building a RAG style system, you want to feed all of your content through an embedding model. That model itself is also potentially something that can be compromised. And I'm curious what you're seeing as some of the threats that are in the ecosystem around that portion of the life cycle of these AI systems.
[00:41:45] Kasimir Schulz:
Yeah. So, actually, so far, it's been fairly unexplored. Though I we actually have a white paper coming out in the next month or so, really exploring a bunch of new attacks about that. So I'll definitely make sure to link you that paper, but I can give a few little kinda hints and things ahead of time. So there's quite a few attacks that can happen there. For example, one of the things that you can do is that there might be certain words that look the same to a person, seem the same to a person, but the embeddings can be wildly different. Or you can have a embedding model that has a different understanding than the underlying LLM. So for example, with a prompt injection detector or spam detector, it might use one set of embeddings for the guardrail model, but another set for the underlying model. So what happens if you drive the embeddings apart? You know, you can really start doing a lot of interesting attacks that way. No. It's
[00:42:42] Tobias Macey:
definitely an interesting aspect of which just makes it so much more challenging to do the analysis and evaluation of what models do I use, how do I build my system, what is the architecture, which probably leaves a lot of people to just throw up their hands and say, I'll just throw money at somebody who I think is doing it the right way.
[00:43:00] Kasimir Schulz:
Yeah. Exactly.
[00:43:03] Tobias Macey:
And in your experience of working in this space, in particular, the work you've been doing around the shadow logic and shadow genes analysis, what are some of the most interesting or innovative or unexpected ways that you've seen the knowledge of model ancestry applied both for offensive and defensive uses?
[00:43:23] Kasimir Schulz:
Yeah. So I think for offensive, it was definitely being able to attack the camera the way that we did. Just being able to generate everything in a nice easy way and then just having 20% of it go over. That was awesome. On the defensive side, though, definitely what I was talking about with you can fine tune your defenses for it. Yeah. I would say that's, you know, the the main thing there is just being able to fine tune any defenses that you want for a specific model. And in your work
[00:43:52] Tobias Macey:
of exploring the space of AI and ML security, what are some of the most interesting or unexpected or challenging lessons that you've learned? I learned that,
[00:44:04] Kasimir Schulz:
I do not understand AI and ML, at all, which has been so much fun. And, honestly, I think it's helped me out a lot because all of the we have so many brilliant data scientists, and they build all these awesome models. And because I don't really understand them necessarily all the time, I'll just poke them the wrong way, and it'll all fall apart. And then I get to learn something completely new. So it's been really, really fun experience. But I think a lot of people have some misunderstandings for those models. And I think that's definitely an experience that I want a lot of other people to have too, especially our data scientists too. Just because sometimes I'll poke something in the wrong way. I won't understand why the output happened that way. I'll go to the data scientist. They'll be like, oh, we don't know either. And then we actually look at it, and we understand some fundamental piece of the model that actually has a huge security impact that we never even considered before just because so many of us treat these AI models as just black boxes and don't try to understand them. Where in reality, they are black boxes and hard to understand, but not to the level that people expect.
[00:45:15] Tobias Macey:
And for teams and individuals and organizations that are investing in the space of ML and AI systems, which is most of them at this point, What are some of the ways that you suggest they keep up to date with the wide and broadening set of attack vectors and threats that are in the ecosystem and means to mitigate and guard against them?
[00:45:41] Kasimir Schulz:
That's always a tough one. That's also a tough one for traditional security. There's going to be certain companies that publish certain work. So we always any attacks that we find, once we actually go through and report them, we publish them. So there's always certain research or blogs that you can go to to read that. On top of that, just staying on top of who are the people who tend to repost the news for you. You might not have to go and find everything, but there might be somebody on x or Mastodon or Blue Sky who will actually go through and post all the news feeds. Let them do the work, and then you can kinda look through.
But it's definitely what I recommend is find the data sources that you are willing to keep up with and try to get as many of those data sources being ones that take in other data sources and kind of filter it down for you. You're going to miss stuff. You're always gonna miss stuff, but that gets you the most important stuff, especially if you have enough data sources. Because once you see, you know, 10 of the people all posted the same exact link, that's probably something I should read rather than just one of them posting something.
[00:46:48] Tobias Macey:
Yeah. One of the newsletters that I've found useful for infosec as well well as AI is the TLDR family of newsletters, and I can add a link in the show notes for that.
[00:46:58] Kasimir Schulz:
Yeah. It's a it's a great one. I'm actually subscribed to that as well. It's part of the reason why I did the MCP blog because I started seeing it a little bit more, and then, all of a sudden, I saw that 10 of the links, not even the AI portion were all titled MCP something. So I was like, okay. I need to actually really look into this.
[00:47:17] Tobias Macey:
Absolutely. Are there any other aspects of the work that you've been doing on shadow logic and shadow genes and the ways that that can and should inform the selection and application of ML models that we didn't discuss yet that you'd like to cover before we close out the show? I mean, the main thing I wanna talk about there is just make sure you scan your models and make sure you're aware of what you're actually using.
[00:47:41] Kasimir Schulz:
Be that shadow genes, be that anything else, but that's just something that's really important. Don't just blindly download and run.
[00:47:50] Tobias Macey:
Absolutely. Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you're doing and the rest of Hidden Layer, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gaps in the tooling technology or human training that are available for AI systems today.
[00:48:08] Kasimir Schulz:
Yes. I think the biggest gap is definitely the learning portion of it. I think we need to do a better job at informing the developers who are actually gonna be implementing it everywhere. Just what are the attacks that are out there? There's already an issue of that with a lot of, developers don't have security experience. I think when I went to I did my undergrad, and I only had one class that was in security before I got into grad school even though I did computer science. So it's it's not hard to attack these systems. So it shouldn't be hard to give, you know, an hour class here or there, make something free available so people can hack away at it, and just learn about what you need to actually look out for. Because just having even if you don't understand it, even if you can't fully do it, just knowing that it's there is going to make you a lot better at securing it.
[00:49:06] Tobias Macey:
And, I guess, on that point too, what is your opinion on the potential for these AI systems to also act in the guidance portion of development and deployment to say, hey. You really shouldn't do that there, or don't forget to check on this thing over there.
[00:49:23] Kasimir Schulz:
I am the first to say that I think AI has sped me up a lot in my day to day work, but you also have to be very careful about how you use it. So a lot of people are using AI to do all of the coding for them, and that's going to start causing a lot of problems down the road, I think. The way that I use it is I use it if I have to do rapid prototyping. If I understand how something works, for example, I think maybe once a month, I'll ask it, write a Python snippet to find all the files recursively of this file type in their directory. It's stuff that I used to have to stack overflow and then modify, but I understand the core concept of how it works. So what it lets me do is because I understand it, it's something I can easily test.
I'm not wasting time trying to debug the code that it made, not wasting time, you know, potentially debugging an issue down the road. So it's really, really amazing for those simple one off things, the prototyping, the looking up documentation, anything that you can very quickly verify is working correctly.
[00:50:28] Tobias Macey:
Absolutely. Yeah. Trust but verify.
[00:50:31] Kasimir Schulz:
Exactly.
[00:50:32] Tobias Macey:
Alright. Well, thank you very much for taking the time today to join me and share the work that you and your team are doing on the Shadow Logic and Shadow Genes work and just giving us an overview of the threat landscape for these AI systems. It's a definitely very important aspect of the work that is being done and something that everybody should definitely be paying a lot of attention to and probably more than they already do. So thank you for all the time and energy that you're putting into helping to surface some of these attack vectors and ways to mitigate them. And I hope you enjoy the rest of your day. Thank you too. And thank you so much for having me on the podcast today.
[00:51:14] Tobias Macey:
Thank you for listening. And don't forget to check out our other shows, the Data Engineering Podcast, which covers the latest in modern data management, and podcast dot in it, which covers the Python language, its community, and the innovative ways it is being used. You can visit the site at the machinelearningpodcast.com to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hoststhemachinelearningpodcast dot com with your story. To help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.
Introduction to AI Engineering Podcast
Interview with Casimir Schultz
Security Challenges in AI Systems
Threat Landscape for ML and AI Systems
Attack Surfaces in Predictive vs Generative Models
Foundation Models and Security Risks
Shadow Logic and Shadow Genes
Detection and Exploitation of Compromised Models
Organizational Maturity and Security in AI Deployment
Offensive and Defensive Uses of Model Ancestry
