Stochastic thermodynamics of Machine Learning : the inevitable cost of neural networks

Details

Supervisors

Delvenne, Jean-Charles

Faculty

Ecole polytechnique de Louvain

Degree label

Master [120] en science des données, orientation technologies de l'information, à finalité spécialisée

Abstract

Using the formalism of stochastic thermodynamics to describe how entropy is produced and flows in a system, this work will describe how a trained neural network unavoidably emits heat when it is run. The two major concepts involved are the Landauer cost, taking its origin from the Landauer principle and which does not depend on the wiring of the circuit, and the mismatch cost, which corresponds to the loss of energy incurred when the input distribution is not the most fitting to the circuit. It will be proven that highly correlated input variables result in higher energetic cost, while uncorrelated variables minimize such cost. While these quantities constitute a lower bound to the entropy flow, a maximal bound to the mismatch cost is proposed. The results obtained on the Landauer and mismatch costs are then tested on a trained model, to which are given correlated and uncorrelated data. This will offer an insight on how to treat data when minimizing the energetic expenses of a trained neural network.