That is the question
-
We have seen numerous articles on AI used for code completion. Now there is complete module generation. A enterprising researcher noted that besides extra code that did nothing, the module appeared to meet the requested specifications. Except for one strange thing. It hallucinated a GIT include of a dependent module's code. Now the included module was not needed and appeared in a section of code that contributed nothing to the actual function of the module. If you ignore the error of a GIT module include not found, the module compiled cleanly and functioned according to spec. This researcher wondered if this was a one-off hallucination or if it was repeatable. He/she coded a simple do nothing stub and placed it in GIT. To the researcher's surprise the GIT module was downloaded over 15,000 times in one month. The researcher then began looking for dependencies in popular commercial products and found it mentioned in several commercial products. This is a really dangerous way for malware to get a foothold in commercial products. this inspired the following (with apologies to William Shakespeare) To code or not to code, that is the question. Whether 'tis nobler to the mind to suffer the slings and arrows of outrageous AI hallucinations or to take up arms against a sea of idiots pushing AI coded modules; and by opposing get labeled a luddite. To be passed over, laid off, no more a member of a team; replaced by the uncaring algorithms churning out incomprehensible logic. The heart-ache and the thousand natural shocks that the flesh is heir to; 'tis a consummation. A marriage made in hell, the bonding of machine and man, for in that fevered union who knows what twisted logic may come. When we have shuffled off this mortal project. When we have born the whips and pangs of QA. The oppressive management scorn, the insolence of the office, the spurns of fellow developers, the legal EULA absolutions of blame, the patient merit of a mentor's frustrated sigh. When might we take time time to document, to grunt and sweat under a weary life. But that dread of something after the project. The undiscovered requirements that no developers have returned from. Rather than bear these ills we fly to other projects to hid from the ills thrust upon us. Thus this spike of conscience makes cowards of us all. The IPOS of great worth, the SPACs of driven financials, and enterprises of great pith and moment, with this regard their current turn awry, and lose the name of action.
-
We have seen numerous articles on AI used for code completion. Now there is complete module generation. A enterprising researcher noted that besides extra code that did nothing, the module appeared to meet the requested specifications. Except for one strange thing. It hallucinated a GIT include of a dependent module's code. Now the included module was not needed and appeared in a section of code that contributed nothing to the actual function of the module. If you ignore the error of a GIT module include not found, the module compiled cleanly and functioned according to spec. This researcher wondered if this was a one-off hallucination or if it was repeatable. He/she coded a simple do nothing stub and placed it in GIT. To the researcher's surprise the GIT module was downloaded over 15,000 times in one month. The researcher then began looking for dependencies in popular commercial products and found it mentioned in several commercial products. This is a really dangerous way for malware to get a foothold in commercial products. this inspired the following (with apologies to William Shakespeare) To code or not to code, that is the question. Whether 'tis nobler to the mind to suffer the slings and arrows of outrageous AI hallucinations or to take up arms against a sea of idiots pushing AI coded modules; and by opposing get labeled a luddite. To be passed over, laid off, no more a member of a team; replaced by the uncaring algorithms churning out incomprehensible logic. The heart-ache and the thousand natural shocks that the flesh is heir to; 'tis a consummation. A marriage made in hell, the bonding of machine and man, for in that fevered union who knows what twisted logic may come. When we have shuffled off this mortal project. When we have born the whips and pangs of QA. The oppressive management scorn, the insolence of the office, the spurns of fellow developers, the legal EULA absolutions of blame, the patient merit of a mentor's frustrated sigh. When might we take time time to document, to grunt and sweat under a weary life. But that dread of something after the project. The undiscovered requirements that no developers have returned from. Rather than bear these ills we fly to other projects to hid from the ills thrust upon us. Thus this spike of conscience makes cowards of us all. The IPOS of great worth, the SPACs of driven financials, and enterprises of great pith and moment, with this regard their current turn awry, and lose the name of action.
That's quite frightening. Yet one more thing to be worried about in the world of software development.
The difficult we do right away... ...the impossible takes slightly longer.
-
We have seen numerous articles on AI used for code completion. Now there is complete module generation. A enterprising researcher noted that besides extra code that did nothing, the module appeared to meet the requested specifications. Except for one strange thing. It hallucinated a GIT include of a dependent module's code. Now the included module was not needed and appeared in a section of code that contributed nothing to the actual function of the module. If you ignore the error of a GIT module include not found, the module compiled cleanly and functioned according to spec. This researcher wondered if this was a one-off hallucination or if it was repeatable. He/she coded a simple do nothing stub and placed it in GIT. To the researcher's surprise the GIT module was downloaded over 15,000 times in one month. The researcher then began looking for dependencies in popular commercial products and found it mentioned in several commercial products. This is a really dangerous way for malware to get a foothold in commercial products. this inspired the following (with apologies to William Shakespeare) To code or not to code, that is the question. Whether 'tis nobler to the mind to suffer the slings and arrows of outrageous AI hallucinations or to take up arms against a sea of idiots pushing AI coded modules; and by opposing get labeled a luddite. To be passed over, laid off, no more a member of a team; replaced by the uncaring algorithms churning out incomprehensible logic. The heart-ache and the thousand natural shocks that the flesh is heir to; 'tis a consummation. A marriage made in hell, the bonding of machine and man, for in that fevered union who knows what twisted logic may come. When we have shuffled off this mortal project. When we have born the whips and pangs of QA. The oppressive management scorn, the insolence of the office, the spurns of fellow developers, the legal EULA absolutions of blame, the patient merit of a mentor's frustrated sigh. When might we take time time to document, to grunt and sweat under a weary life. But that dread of something after the project. The undiscovered requirements that no developers have returned from. Rather than bear these ills we fly to other projects to hid from the ills thrust upon us. Thus this spike of conscience makes cowards of us all. The IPOS of great worth, the SPACs of driven financials, and enterprises of great pith and moment, with this regard their current turn awry, and lose the name of action.
Hang on a sec.
Quote:
He/she coded a simple do nothing stub and placed it in GIT. To the researcher's surprise the GIT module was downloaded over 15,000 times in one month. The researcher then began looking for dependencies in popular commercial products and found it mentioned in several commercial products
I assume this means: S/he created a git repo. The code in that repo was "downloaded" and started to be included in commercial products. Maybe I'm missing something here, but is there any details on what "downloaded" means? Forked repo? Zip of code downloaded? Since it says he noted commercial products had dependencies on his code, I assume this means the code was actually packaged in a PyPi / Nuget / npm etc package and that was what was downloaded (by developers and the as part of the installation of the commercial products). The question that then comes to mind is: How did he find the dependencies of commercial products? I'm assuming he / she didn't go around randomly cracking private git repos to check out ISVs' code, so I assume it's more about installing products and seeing what gets sucked down. Plus there is the "dependency of a dependency of a ..." thing. If he got his package made a dependency of a single, vaguely popular package, then he's in like the proverbial Trojan Horse. It's a great story but I am dying for the details!
cheers Chris Maunder
-
We have seen numerous articles on AI used for code completion. Now there is complete module generation. A enterprising researcher noted that besides extra code that did nothing, the module appeared to meet the requested specifications. Except for one strange thing. It hallucinated a GIT include of a dependent module's code. Now the included module was not needed and appeared in a section of code that contributed nothing to the actual function of the module. If you ignore the error of a GIT module include not found, the module compiled cleanly and functioned according to spec. This researcher wondered if this was a one-off hallucination or if it was repeatable. He/she coded a simple do nothing stub and placed it in GIT. To the researcher's surprise the GIT module was downloaded over 15,000 times in one month. The researcher then began looking for dependencies in popular commercial products and found it mentioned in several commercial products. This is a really dangerous way for malware to get a foothold in commercial products. this inspired the following (with apologies to William Shakespeare) To code or not to code, that is the question. Whether 'tis nobler to the mind to suffer the slings and arrows of outrageous AI hallucinations or to take up arms against a sea of idiots pushing AI coded modules; and by opposing get labeled a luddite. To be passed over, laid off, no more a member of a team; replaced by the uncaring algorithms churning out incomprehensible logic. The heart-ache and the thousand natural shocks that the flesh is heir to; 'tis a consummation. A marriage made in hell, the bonding of machine and man, for in that fevered union who knows what twisted logic may come. When we have shuffled off this mortal project. When we have born the whips and pangs of QA. The oppressive management scorn, the insolence of the office, the spurns of fellow developers, the legal EULA absolutions of blame, the patient merit of a mentor's frustrated sigh. When might we take time time to document, to grunt and sweat under a weary life. But that dread of something after the project. The undiscovered requirements that no developers have returned from. Rather than bear these ills we fly to other projects to hid from the ills thrust upon us. Thus this spike of conscience makes cowards of us all. The IPOS of great worth, the SPACs of driven financials, and enterprises of great pith and moment, with this regard their current turn awry, and lose the name of action.
-
Hang on a sec.
Quote:
He/she coded a simple do nothing stub and placed it in GIT. To the researcher's surprise the GIT module was downloaded over 15,000 times in one month. The researcher then began looking for dependencies in popular commercial products and found it mentioned in several commercial products
I assume this means: S/he created a git repo. The code in that repo was "downloaded" and started to be included in commercial products. Maybe I'm missing something here, but is there any details on what "downloaded" means? Forked repo? Zip of code downloaded? Since it says he noted commercial products had dependencies on his code, I assume this means the code was actually packaged in a PyPi / Nuget / npm etc package and that was what was downloaded (by developers and the as part of the installation of the commercial products). The question that then comes to mind is: How did he find the dependencies of commercial products? I'm assuming he / she didn't go around randomly cracking private git repos to check out ISVs' code, so I assume it's more about installing products and seeing what gets sucked down. Plus there is the "dependency of a dependency of a ..." thing. If he got his package made a dependency of a single, vaguely popular package, then he's in like the proverbial Trojan Horse. It's a great story but I am dying for the details!
cheers Chris Maunder
Okay, finally located the article. Yes it was a Python/pip inclusion problem. Here is the original article to correct for my blurry memory. AI bots hallucinate software packages and devs download them • The Register[^]
