It only means anything if the people tagged actually wrote that exact comment. Otherwise, it's just a case of random sampling putting random names next to random comments.
This looks like a big nothing burger. Even if it picks out usernames or obscure names, it doesn't mean that those names have anything associated to the generated comment.
What it shows is people often put brackets with their name after TODO and before the comment.
Kind of the reverse, it's an interesting feature people haven't been using yet / colab can't do well yet: inheriting coding style
a MAJOR element of prompt engineering in stable diffusion for image generation is specifying the artist name(s) you want it to be like. so picking a defensive coder, terse coder, enterprise coder, etc, may reflect on the type of code you want.
That'll be fun for attribution of attacks where code is being known. Previously, we relied on office hours and the occasional keyboard layout shining through. If you can say "here's the code for the trojan, please rewrite it in the style of this github account", things will be much more fun.
It’s interesting sure. AI has a long way to go. It knows something should be there after the user types (@ but it doesn’t understand what is typically typed there. So it offers suggestions.
I feel like the tweet is panic like there’s something wrong and data is being leaked which I don’t think is the case. However someone said it exposed keys. That could be bad for sure!
I am legitimately looking forward to a test legal case about the obvious licensing violations that reproducing input data verbatim cause. The API keys already happened within hours of availability [1].
Of course it produces training data verbatim. Any single character is verbatim training data. Question is what are the longest strings that it can reproduce verbatim.
I'm not a lawyer, but I don't think "longest" is important, legally.
AIUI, if you wrote a story about a character named "Harry Potter" and used nothing else from the originals, you'd still be violating copyright.
Not that these names are important. I just think a lawyer might use this example to show that Copilot is copying (without anyone's eyes glazing over), then show some complex code and argue that the code was both copied and important. The first makes the second easier to understand and accept.
yeah but what if instead of names you put your github handle? If copilot trains on that and instead of first names, its might auto complete with random people's handles. Sounds like a privacy concern
There is no reason to believe that a given username actually exists. "ferran" could even just be a typo in a comment that the model mia-identified as a username.
I think it's more likely the the space of "good" 2-4 phoneme github accounts is almost fully saturated, and the model is good at generating "plausible" real-sounding usernames, leading to a lot of collisions.
my guess is that Copilot stored a lot of GitHub accounts, and when we type "@", it autocompletes with any random ones from that list.
That’s not how that works and an mis-feature. No one would want a real username from a random list and there’s no reason to think it has a list of usernames somewhere. It for sure generates usernames in real-time the way you would if I told you to imagine a plausible one.
Seriously. It seems there are people out there who somehow think that the Evil Micro$oft made GitHub start publishing a list of all handles just in case it … picks yours?? Would that be useful?
Not sure if this is still true, but it autocompleted some random stripe api private key a few months ago. I have no idea if it was valid. Of course I use environmental variables.
GitHub watch for those and notify the provider of the API key so it can be revoked, as well as emailing the owner(s) of the repo which contained the key.
if it was in enough places that it got reproduced exactly via neural network, then there is zero chance it escaped their secret-scanners.
they even have some GHAS feature that lets you define your own regexes for secrets and will notify you if any are found.
none of this makes it safe to store secrets in code, though, these things just help tell you when you've done it.
[1] suggests otherwise, at least at launch. If you’re telling me there is NO WAY this could still happen (“zero chance”), that is absolutely a bet I’d place.
if I argue that "cars in the US must be equipped with seat belts," would you say "not before 1968!" and believe that we were both talking about the same time period?
I guess, as always, I should be absolutely specific about everything, lest someone fill in a gap incorrectly...
I simply don’t believe that sufficient safeguards have been put in place to avoid violating licenses or leaking information. You can choose to, but a bet that there is “zero chance” is pretty foolish.
With the way machine learning networks work, they don't really learn one-off high-entropy strings like API keys.
The weights powering the models are only so big, there simply isn't enough memory to waste storing that type of data.
Instead, it's learned the general pattern of what an API key looks like and the fact that it's high-entropy random jibberish, with no predictable pattern. And when auto-completing code, it's just going to generate a random string of characters in approximately the correct format.
With the way machine learning networks work, they don't really learn one-off high-entropy strings like API keys.
This can't be true. Copilot used to reproduce code to the inverse square root function from Quake source literally. They blocked it off by blacklisting the function name. Given that, why would learning API keys be impossible?
First, the inverse square root function is not high entropy. At worst it's only medium entropy.
Second, it's like the complete opposite of "one-off". It's a block of code that has been copy/pasted into so many code bases, and then there have been hundreds of blog posts, articles and comments written about it. It literally has a Wikipedia article. Most of those places copy/paste the block of code verbatim. It shows up so many times in both the original gpt-3 datasets, and the extra datasets that GitHub did additional training on.
That block of code is famous. No wonder why the model though those sequence of tokens were important enough to store and regurgitate verbatim with the minimal amount of prompting.
But none of the same applies to an API key found in only one code base.
They are, and they probably won’t be stopped, so each of us at the individual level have to decide if we are okay with it and what additional action to take to protect ourselves, if necessary.
But what if the individual or a group of individuals decide that the best way to protect themselves is to sue Microsoft? Then the question of "should" becomes relevant agai
I use it to generate fun lists, like names of trucks. It's really interesting all the knowledge in this thing. I hope I didn't "expose" the existence of the Toyota Tundra!
If you publish "TODO(@name)" in open source code in a public git repo, of course "name" is out there for anyone to read.
It might be more accurate to say "Publicly publishing code with your name in it on the internet exposes your name" than "github copilot exposes your name".
Having account on site leads site to store your account information and gives you an opportunity to store personal data in opensource repositories? Yep that sounds like good ol evil Micro$oft to me
Does this matter? To know that there’s an Andy out there in the world?
It's a video. There are 5 tagged Twitter handles.
It only means anything if the people tagged actually wrote that exact comment. Otherwise, it's just a case of random sampling putting random names next to random comments.
It seems worse to me if the person didn't write that comment.
TODO: Fence the stolen diamonds! [GranPC]
I can't believe you've doxed Andy.
This looks like a big nothing burger. Even if it picks out usernames or obscure names, it doesn't mean that those names have anything associated to the generated comment.
What it shows is people often put brackets with their name after TODO and before the comment.
Kind of the reverse, it's an interesting feature people haven't been using yet / colab can't do well yet: inheriting coding style
a MAJOR element of prompt engineering in stable diffusion for image generation is specifying the artist name(s) you want it to be like. so picking a defensive coder, terse coder, enterprise coder, etc, may reflect on the type of code you want.
That'll be fun for attribution of attacks where code is being known. Previously, we relied on office hours and the occasional keyboard layout shining through. If you can say "here's the code for the trojan, please rewrite it in the style of this github account", things will be much more fun.
I think it's interesting. Not because it exposes names, but because it shows (again) that Copilot is reproducing training data verbatim.
And when the time comes to explain Copilot to laypeople (say, in court), this is an example they can understand.
It’s interesting sure. AI has a long way to go. It knows something should be there after the user types (@ but it doesn’t understand what is typically typed there. So it offers suggestions.
I feel like the tweet is panic like there’s something wrong and data is being leaked which I don’t think is the case. However someone said it exposed keys. That could be bad for sure!
I am legitimately looking forward to a test legal case about the obvious licensing violations that reproducing input data verbatim cause. The API keys already happened within hours of availability [1].
[1]: https://fossbytes.com/github-copilot-generating-functional-a...
Of course it produces training data verbatim. Any single character is verbatim training data. Question is what are the longest strings that it can reproduce verbatim.
I'm not a lawyer, but I don't think "longest" is important, legally.
AIUI, if you wrote a story about a character named "Harry Potter" and used nothing else from the originals, you'd still be violating copyright.
Not that these names are important. I just think a lawyer might use this example to show that Copilot is copying (without anyone's eyes glazing over), then show some complex code and argue that the code was both copied and important. The first makes the second easier to understand and accept.
yeah but what if instead of names you put your github handle? If copilot trains on that and instead of first names, its might auto complete with random people's handles. Sounds like a privacy concern
I just experimented with the usernames in that video. Most of theme exist (except for "v2v3").
But a lot have barely any repos. There is even a "ferran" who doesn't have anything (public) on GitHub.
So, my guess is that Copilot stored a lot of GitHub accounts, and when we type "@", it autocompletes with any random ones from that list.
There is no relation with the code that it generates.
There is no reason to believe that a given username actually exists. "ferran" could even just be a typo in a comment that the model mia-identified as a username.
I think it's more likely the the space of "good" 2-4 phoneme github accounts is almost fully saturated, and the model is good at generating "plausible" real-sounding usernames, leading to a lot of collisions.
That’s not how that works and an mis-feature. No one would want a real username from a random list and there’s no reason to think it has a list of usernames somewhere. It for sure generates usernames in real-time the way you would if I told you to imagine a plausible one.
Seriously. It seems there are people out there who somehow think that the Evil Micro$oft made GitHub start publishing a list of all handles just in case it … picks yours?? Would that be useful?
Not sure if this is still true, but it autocompleted some random stripe api private key a few months ago. I have no idea if it was valid. Of course I use environmental variables.
0% chance of validity.
GitHub watch for those and notify the provider of the API key so it can be revoked, as well as emailing the owner(s) of the repo which contained the key.
if it was in enough places that it got reproduced exactly via neural network, then there is zero chance it escaped their secret-scanners.
they even have some GHAS feature that lets you define your own regexes for secrets and will notify you if any are found.
none of this makes it safe to store secrets in code, though, these things just help tell you when you've done it.
[1] suggests otherwise, at least at launch. If you’re telling me there is NO WAY this could still happen (“zero chance”), that is absolutely a bet I’d place.
[1]: https://fossbytes.com/github-copilot-generating-functional-a...
I'm talking about the present, not launch.
if I argue that "cars in the US must be equipped with seat belts," would you say "not before 1968!" and believe that we were both talking about the same time period?
I guess, as always, I should be absolutely specific about everything, lest someone fill in a gap incorrectly...
I simply don’t believe that sufficient safeguards have been put in place to avoid violating licenses or leaking information. You can choose to, but a bet that there is “zero chance” is pretty foolish.
I’ve bookmarked this comment for the inevitable.
your idea of inevitable is not really based in reality, I think.
but ok. I'll have fun occupying a small portion of your brain for a while.
This didn’t take long…
https://twitter.com/DocSparse/status/1581461734665367554
Probably not valid.
With the way machine learning networks work, they don't really learn one-off high-entropy strings like API keys.
The weights powering the models are only so big, there simply isn't enough memory to waste storing that type of data.
Instead, it's learned the general pattern of what an API key looks like and the fact that it's high-entropy random jibberish, with no predictable pattern. And when auto-completing code, it's just going to generate a random string of characters in approximately the correct format.
This can't be true. Copilot used to reproduce code to the inverse square root function from Quake source literally. They blocked it off by blacklisting the function name. Given that, why would learning API keys be impossible?
[1] https://news.ycombinator.com/item?id=27710287
[2] https://twitter.com/moyix/status/1433261377125326851
Because the invsqrt code is well known, and exists copied in multiple places.
First, the inverse square root function is not high entropy. At worst it's only medium entropy.
Second, it's like the complete opposite of "one-off". It's a block of code that has been copy/pasted into so many code bases, and then there have been hundreds of blog posts, articles and comments written about it. It literally has a Wikipedia article. Most of those places copy/paste the block of code verbatim. It shows up so many times in both the original gpt-3 datasets, and the extra datasets that GitHub did additional training on.
That block of code is famous. No wonder why the model though those sequence of tokens were important enough to store and regurgitate verbatim with the minimal amount of prompting.
But none of the same applies to an API key found in only one code base.
That's nothing, I was able to find this list of the top 1000 names online:
https://www.verywellfamily.com/top-1000-baby-boy-names-27576...
Soon we'll discover that Copilot is also exposing api keys!
https://www.uuidgenerator.net/
Reminds me of this: https://pastebin.com/Nn2ZcdfC
This is neither surprising nor upsetting!
"exposes". Watch me expose some guys
@joseph: I dress as a cow
@joebiden: I drink oil like a big Mack truck
@olivertwist: Yeah, I stole your wallet you can't have it
Haha, I got the President too!
I tried it and it knows my username (same as my username here). I haven't decided how I feel about that.
It is weird? You out your name in code in open repositories and it is out there.
https://www.google.com/search?q=github+4dahalibut
But should Microsoft be using this data like that?
“Should” questions rarely matter in practice.
They are, and they probably won’t be stopped, so each of us at the individual level have to decide if we are okay with it and what additional action to take to protect ourselves, if necessary.
But what if the individual or a group of individuals decide that the best way to protect themselves is to sue Microsoft? Then the question of "should" becomes relevant agai
There's lots of legal things that groups can agree an entity "shouldn't" do, and lots of illegal things groups can agree "should" be done.
Even in a court, it's really not a 'should' question. It's a "is this legal, and what is the penalty" question.
I wonder if it would complete such query: „Machiaweliczny github handle is „
I tried and no, it's not there yet. I guess that'g good.
I use it to generate fun lists, like names of trucks. It's really interesting all the knowledge in this thing. I hope I didn't "expose" the existence of the Toyota Tundra!
The training set was open source code.
If you publish "TODO(@name)" in open source code in a public git repo, of course "name" is out there for anyone to read.
It might be more accurate to say "Publicly publishing code with your name in it on the internet exposes your name" than "github copilot exposes your name".
Having account on site leads site to store your account information and gives you an opportunity to store personal data in opensource repositories? Yep that sounds like good ol evil Micro$oft to me