Is Machine Learning Random? And Why Should You Care?
Let’s start with a common problem that people have, especially when they are just starting out with machine learning.
It has to do with running the same program multiple times, and getting different results.
Machine learning gurus on the web are often flooded with queries from frustrated users who are trying to work with a particular set of algorithms, and don’t understand why the program isn’t providing a consistent result.
Machine learning makes prolific use of randomness. That happens in a lot of different ways. You might see that your training data gets selected at random, or that there’s a random element within the algorithm itself.
In a very fundamental sense, that’s part of the joy of machine learning and digital neural networks.
In other words, if you wanted only purely deterministic programmed results, you wouldn’t really need machine learning at all, in some cases. The other day, we heard none other than Stephen Wolfram talking about computational irreducibility and making that sort of point.
If you want more insight on randomness within modern computer science, along with some hard math and groundbreaking research references, Ted Pyne has the goods.
Starting with the utility of randomness for things like sort and run scenarios, he explains a conjecture that would suggest you can pursue building deterministic algorithms that can do the same work as those with random underpinning. (For use cases, he mentions cryptography, communications, and sublinear algorithms.)
In support of this theory, he talks about the nature of small circuits: we’ll leave this part to the speaker himself – as you watch, think about the logic underneath these suppositions.
Pyne references Nisan-Wigderson (pseudorandom generator) and Impagliazzo-Wigderson (P=BPP Paper) in looking at hardness and randomness in machine learning programs.
He also talks about derandomizing space-efficient algorithms using the work of Klivans-VanMelkebeek.
One tip for getting more involved in this sort of research is to understand the use of data structures like matrices and eigenvectors.
To do that, you can go back and listen to some of the other compelling videos around the IAA event, and get more of a groundbase of knowledge. Then when you come back and listen to this one again, it might make more sense.
Pyne is basically talking about testing new systems in new ways. There’s something to be said, in this part of the digital era, for new wine in new wineskins, for throwing out old methods and seeing what’s possible with new tools. Because these days, we have a lot of them!
As for randomness, knowing that it has its place in machine learning will help you to become a more tech-savvy user as we continue to see AI offer us more self-service options in building applications.
