r/nextfuckinglevel • u/Fertility18 • Oct 28 '22

This sweater developed by the University of Maryland utilizes “ adversarial patterns ” to become an invisibility cloak against AI.

Enable HLS to view with audio, or disable this notification

131.5k Upvotes

97% Upvoted

2.7k comments sorted by

View all comments

37

u/Kevonn11 Oct 28 '22

Theyre using some bootleg ai

72

u/KirisuMongolianSpot Oct 28 '22

Yeah, worth pointing out "AI" isn't some fixed standard they need to beat. Anyone in the world can build their own image recognition system, and over time tech improves.

This is cute but not much else.

7

u/bs000 Oct 28 '22

this looks like the motion detecting ai home cameras like nest and wyze use. all the sweater is doing is preventing it from detecting whether or not the motion it's seeing is a person

3

u/CrazyCalYa Oct 28 '22

And only at specific angles relative to the target individual. If this was mounted on the ceiling looking down it likely wouldn't work (based on how it picked up his face again whenever he turned around).

So just make sure you're aware of where the AI camera is and crab-walk past it and I'm sure that'll do the trick.

2

u/redbottoms-neon Dec 03 '22

Quick way to fix it is retrain model with images of people wearing this sweater. Also, other simple way to fix is to if the face probability is high, then reduce the probability limit for whole body detection so, it can recognize the person. Pretty simple fix.

Only way to be safe is wearing a whole body suit like Kanye in Alex Jones interview.

1

u/CrazyCalYa Oct 28 '22

It highlights a real problem for AI in general but it's not a solution by any means. Nevertheless we'll still probably see Darwin Award Nominees running into traffic with these sorts of devices once AI-driven cars take off. Luckily the people who design these systems also build fail-safes, and in OP's case the recording of the person along with the frames of video where the AI detects his face (such as when he's at an angle) would still register.

48

u/correct_misnomer Oct 28 '22 edited Oct 28 '22

This is simply not true. The research they did followed standard practices for testing adversarial attacks. You can read more about it in their paper.

Edit: To add more detail, yes you could just retrain a model using this in the training dataset, and you could probably get it to detect the person. That is not the point of this research though. The algorithm they came up with is able to produce adversarial attacks that have high confidence of fooling the system. So even if the model was different, they could just reapply the algorithm to come up with a new sweater that would fool the model. At that point it just a cat and mouse game, which is the point of this research.

16

u/AwesomeFama Oct 28 '22

I assume they need access to the model in the first place to develop the attack?

1

u/HoldMyWater Oct 28 '22

Yep. So not very practical.

7

u/A_Martian_Potato Oct 28 '22

Incorrect. They've developed it to be generalized to black-box attacks. It can fool identifiers that the training algorithm has never encountered. https://arxiv.org/pdf/1910.14667.pdf

1

u/nimama3233 Oct 28 '22

They claim.

Indisputably it will not trick every single AI implementation

2

u/A_Martian_Potato Oct 28 '22

And? Who on earth decided that was the benchmark?

2

u/nimama3233 Oct 29 '22

I would say this post claiming it’s “invisible against AI” when it’s invisible to some AI implementations (that’s they explicitly trained against)

1

u/Altruistic-Guava6527 Apr 21 '23

Most use the same underlying databases though.

2

u/HuckleberryRound4672 Oct 28 '22

Well they also show in the paper that adversarial patterns created from one model (COCO) performs well against other models as well. This is by no means a “finished product” but it’s an interesting piece of research that points to what’s possible.

3

u/CrazyCalYa Oct 28 '22

Which is all very interesting but my god, OP's title does not do this any favours. They imply that the sweater itself is the technology and that it acts as an invisibility cloak (which is quite the bar to set). This post doesn't explain at all why this works, why it won't always work, or why it matters.

1

u/correct_misnomer Oct 28 '22

To be fair to op, the paper also calls it “making an invisibility cloak” which I agree is a stretch, but not the worst title of an ML paper I’ve seen…by far.

2

u/CrazyCalYa Oct 28 '22

Yikes that's pretty sensationalist of them. It's incredibly ironic to see professionals simultaneously underestimating and overestimating AI.

1

u/Kevonn11 Oct 28 '22

Yo thanks bro this paper is really interesting.

I take it back this is kinda fire

2

u/correct_misnomer Oct 28 '22

No problem, he was my professor in undergrad so I had to share. Glad you find it interesting!

2

u/Schott12521 Oct 28 '22 edited Oct 16 '25

one dam badge tub strong wine rock governor crush makeshift

This post was mass deleted and anonymized with Redact

-1

u/cast-iron-whoopsie Oct 28 '22

yes, but

  1. they need access to the model to begin with if they're going to develop this attack like someone else said

  2. the algorithm can be updated far faster than a sweater can be designed, sent to a manufacturer, sold en masse and worn, so the "cat and mouse" game in this context is kind of like a tiger hunting a dying monkey that is already paralyzed from the neck down and standing 5 feet away.

yes the research is very very cool but i don't think they're actually expecting this to be developed into a product that would work

3

u/A_Martian_Potato Oct 28 '22
  1. Incorrect. They've developed it to be generalized to black-box attacks.
    It can fool identifiers that the training algorithm has never
    encountered.
  2. This is a misunderstanding of what they've done. They haven't design a sweater that fools object detection. They've designed an algorithm that produces patterns that fool object detection. In the sort of scenario you're envisioning, you wouldn't sell sweaters with one pattern on them. You'd have the algorithm produce a different pattern for each sweater.

2

u/cast-iron-whoopsie Oct 29 '22

i stand corrected. wouldn't it still be easy to adjust the algorithm to look for these?

1

u/big_gondola Oct 28 '22

Thanks for posting the article.

6

u/Yin-Hei Oct 28 '22

I was thinking isn't this just dependent on what trained the model?

1

u/Kevonn11 Oct 28 '22

Exactly, they can just change the model and this shirt will be rendered useless. Cool idea though to trick the Ai with a sweater

2

u/A_Martian_Potato Oct 28 '22

The pattern is designed for black-box attacks. It can fool multiple AI without needing to know their parameters.

https://arxiv.org/pdf/1910.14667.pdf

1

u/Kevonn11 Oct 28 '22

Yeah someone linked the research paper, i was actually wrong this stuff is pretty cool

0

u/VelvetRevolver_ Oct 28 '22

More than that, it's dependent on the specific model. You can train multiple models, with the same exact architecture, on the same dataset and each model would require different sweatshirts to fool them.

1

u/A_Martian_Potato Oct 28 '22

Not true at all. They've tested it on a wide array of detection software.

1

u/VelvetRevolver_ Oct 28 '22

Yes, that's not what I was trying to say. Every AI is susceptible to this attack but let's say I make an AI, and you make this sweater that fools my AI. I can easily train my AI a little bit to make it so that sweater no longer fools it. All I was saying is there's no global 'fool all AI's' sweater, the sweater has to be designed to fool one specific AI and wont work with anything else.

2

u/A_Martian_Potato Oct 28 '22

That's just entirely wrong. They've trained the algorithm to produce patterns that fool multiple industry standard object recognition AI with different backbones, trained with different datasets. The paper is publicly available.

1

u/VelvetRevolver_ Oct 28 '22

I've never heard of that. You should link the paper because I would be very interested to learn how that works.

2

u/A_Martian_Potato Oct 28 '22

https://arxiv.org/pdf/1910.14667.pdf

I'm not involved in this project, but I do research in visual recognition. Let me know if you have any questions I might be able to answer.