Optimal initialization of neural networks

Abstract

Artificial neural networks have seen a big surge in popularity. The reason for that is the fact that a lot of different areas, such as photo/video – oriented tasks (object detection, object recognition, semantic segmentation, bounding boxes etc.), neural machine translation, optical character recognition, automated driving and many others, are at the moment the best application of such an approach. Moreover, the use of so called “deep” neural networks is getting widely known now. While having both the computational power increase (modern GPUs are helping with the matter) and the access to enormous amounts of data pushes machine learning engineers and researchers to give up the idea of shallow networks (the ones that have few layers).