Abstract: America has a massive railway system. As of 2006, U.S. freight railroads have 140,490 route- miles of standard gauge, but maintaining such a huge system and eliminating any dangers, like reduced track stability and poor drainage, caused by railway ballast degradation require huge amount of labor. The traditional way to quantify the degradation of ballast is to use an index called Fouling Index (FI) through ballast sampling and sieve analysis. However, determining the FI values in lab is very time-consuming and laborious, but with the help of recent development in the field of computer vision, a novel method for a potential machine-vison based ballast inspection system can be employed that can hopefully replace the traditional mechanical method. The new machine-vision approach analyses the images of the in-service ballasts, and then utilizes image segmentation algorithm to get ballast segments. By comparing the segment results and their corresponding FI values, this novel method produces a machine-vision-based index that has the best-fit relation with FI. The implementation details of how this algorithm works are discussed in this report.