Thoughts on research so far

2 minute read

Published:

So I’ve been reading the field of machine learning for some time now, and I must say I’m disappointed. Survey any large conference such as NeurIPS, ICML, or ICLR, and 90% of the papers are backpropagation with some extra code sprinkles. Now I can understand that, it’s a powerful technique that is general enough to solve many tasks, and I like that too. But there is a serious lack of deep thought happening here, which is the typical technique when solving any problem. It’s like AI is already autonomising our problem solving technique by tempting researchers into a gold rush scenario, leaving everyone with heat stroke from the searing heat of our GPUs. On the other hand it’s quite nice, you only have to read 10% papers that might contain original ideas that aren’t fully crackpot.

Ultimately what matters is that you can get an algorithm to do something interesting before your next birthday. But this is already difficult; there is a natural argument for intelligent computation characterised by very long sequences of mathematical / logical operations, so there will always be an expectation that results will take time to compute (hence why the mass development of mathematically accelerated hardware is a great investment of the future). I think what matters most is an intuition of these things, rather than sitting around testing one idea after the next, the latter will seriously kill any enthusiasm you have for the subject. Machine learning is still a baby field, and will probably feel that way for all time until even more powerful classes of algorithms have been thoroughly tested. Always remember that no matter how beautiful the theory and squiggles on paper, if it can’t be computed, it’s not a feasible technique and not part of machine intelligence (speed is a large factor of intelligence).

Looking forward, I am hopeful that new ideas will come soon. Even if they aren’t as powerful as the previous technique, it is always great to see uniquely new directions that are comparable in at least some dimensions. It’s easy to criticise, but that doesn’t mean we don’t enjoy the field; very little is solved yet, making it very fun to be part of. Wouldn’t it be a shame if you were born into a time where everything was!

Leave a Comment