Visual and tactual examination of unprocessed breast specimens is the standard for intraoperative surgical margin assessment in the United States. However, this procedure does not provide surgeons or pathologists with microscopic views of the tissue, which makes it difficult to accurately assess margin status or the extent of the disease, especially in non-palpable cases. We use a combination of spectral and polarization macroscopic imaging to optically segment the adipose and collagen tissues thus highlighting regions suspected of containing epithelium in order to facilitate optical microscopy techniques. A small study on five lumpectomy and mastectomy samples showed a sensitivity of 70% ± 20% and specificity of 50% ± 10% for adipose segmentation and a sensitivity of 50% ± 20% and specificity of 50% ± 20% for collagen segmentation. This sensitivity and specificity are sufficient for providing morphological information to the pathologist in order to guide microscopic examination of regions likely to be of clinical significance.