Today, the process of sperm selection for in-vitro fertilization (IVF) is practiced by experienced clinicians using non-quantitative optical microscopy systems. The information present in such techniques is limited while factors integral to conception are dismissed. In this work, we present a method for automated DNA fragmentation analysis using quantitative phase maps of single sperm cells acquired through use of interferometric phase microscopy (IPM). With just below 600 cells from different donors, we were able to achieve precision of 90%, with an area under the receiver operating characteristic curve of 80% and an area under the precision recall curve of 99%. This was achieved through creating a bimodal deep learning model with inputs of both the images taken through IPM and their acquired morphological features. The use of this algorithm in clinics will allow for an automated objective selection of sperm cells that should increase the success rates in IVF.