I think these concerns are why people want to use only very specific AI for very narrow issues.
right now we are used to "generative AI" for some words, including some code, images, even movie previews, but at least 3.5 is kind of "closed". experts like sam altman say chatGPT-4 is a little embarrassing even though most of us find it to be amazing.
https://www.youtube.com/watch?v=GLKoDkbS1Cg
that's a bit hard to follow since clearly smarter tech is coming. when whoever is talking in the video says "superintelligence", he probably means AGI. which seems to mean, roughly, like something where all these narrow AIs are out there, then there is a "meta AI" that can "talk to" many narrow AIs. then coordinate them. and so on.
do we unleash that into manufacturing? geez i hope not. i think maybe the slippery slope is like the chess game. "i have this thing that does xyz way better, but i don't understand how it works .... should I use it?" .... for most students (and people and executives), we basically say "yes" to tools we don't understand all the time. I don't understand how my car works or how a plane works or how my phone GPS works, for example.