As someone who does code, this is a very valid concern you have. A lot of software is being built right now that not a single human actually understands. On some lever it could be possible to understand but that doesn’t change the fact no one does actually understand it.
To be clear, I’m not saying it’s so advanced that we don’t understand it. The problem is it’s such a giant pile of shit that we don’t understand it.
As someone who does code, this is a very valid concern you have. A lot of software is being built right now that not a single human actually understands. On some lever it could be possible to understand but that doesn’t change the fact no one does actually understand it.
To be clear, I’m not saying it’s so advanced that we don’t understand it. The problem is it’s such a giant pile of shit that we don’t understand it.
Especially if the code touches things where the stakes are high.
Integrity of systems e.g. used by dissidents in whatever authoritarian country.
Heavy machines.
Cars, trucks, trains, subways.
Industrial robots - if out of control, their force and speed can easily kill people.
Critical infrastructure for electricity, water, energy, payments, and transport.
Chemical factories.
Medical devices.
This list is expanding because more and more digital systems become enmeshed with the physical world.
Imagine AI systems getting AI-stuxnet’d