In this pilot study, we introduce a machine learning framework to identify relationships between cancer tissue morphology and hormone receptor pathway activation in breast cancer pathology hematoxylin and eosin (H&E)-stained samples. As a proof-of-concept, we focus on predicting clinical estrogen receptor (ER) status—defined as greater than one percent of cells positive for estrogen receptor by immunohistochemistry staining—from spatial arrangement of nuclear features. Our learning pipeline segments nuclei from H&E images, extracts their position, shape and orientation descriptors, and then passes them to a deep neural network to predict ER status. After training on 57 tissue cores of invasive ductal carcinoma (IDC), our pipeline predicted ER status in an independent test set of patient samples (AUC ROC = 0.72, 95%CI = 0.55–0.89, n = 56). This proof of concept shows that machine-derived descriptors of morphologic histology patterns can be correlated to signaling pathway status. Unlike other deep learning approaches to pathology, our system uses deep neural networks to learn spatial relationships between pre-defined biological features, which improves the interpretability of the system and sheds light on the features the neural network uses to predict ER status. Future studies will correlate morphometry to quantitative measures of estrogen receptor status and, ultimately response to hormonal therapy.