Except that's not how it works.
after 6 years of minoring on information theory and neural nets I'm opinioning that it's exacty how it works.
all neural nets are classifiers that divide inputs into n-dimensional space where n is the number of 'correct outputs' it's been trained to 'recognize' ie categorize. what the new batch of nnets are doing is making linear combinations of those, ie. LITERALLY "remixing other people's work" and nothing else. there is absolutely no 'thinking' or 'creating' going on, just copying and interpolating within the vast mountain of training data created by actual artists.
now, it could be claimed that our brain is doing the exact same thing, and partly that would be true. except that it isn't, and our brain is doing something weird called 'thinking' in addition to it, and we have no clue whatsoever where that 'thinking' part comes from. (it's not 'soul' though). even the aforementioned basic model of how nnets classify we kinda ran into by accident, and we don't really know what happens to the weights inside the nnet.
I mean we know of course how we're adjusting the weights, but we don't understand the emergent content of it. in other words, we can't 'read' the 'thought' from the neurons even though we know exactly their state and how we adjusted them. we just treat it as a 'black box' and wait for the output at the other end.
one day we'll definitely crack it, because it's no magic and there's no 'soul' involved in any of it. but today we don't really have a clue where to even begin. people are just fooled by the pretty pictures and language that LOOKS natural, because they don't understand there is zero cognitive processing behind it. chatGPT is not 'thinking' when is spews out that 'speech', it's mechanically combining the texts of OTHER people and generating an output that LOOKS LIKE the linear combination of its training data.
but I wanna repeat that this does not mean 'machines will never be sentient, they just copy things very well'. you know, what data was said to do in star trek. it's just that we're not there, and we have no idea what we should even try to start the journey there. right now it IS just clever mindless copying, but one day will figure out how to make nnets actually think. so that they can create new stuff and not just copy others.