Early and rapid diagnosis of herbicidal activity is essential for herbicide discovery and development. Spectral image analysis can detect plant responses nondestructively even when no visual symptom occurs, and is expected to play a key role in high throughput screening (HTS) of new herbicidal compounds. Therefore, this study was conducted to diagnose herbicidal activities of various herbicides with different modes of action and to analyze the relationship between herbicide modes of action and corresponding spectral responses. Weed plants at an early growth stage were treated with several herbicides with different modes of action and their spectral images including RGB, IR thermal and chlorophyll fluorescence images were acquired at early timings after herbicide treatment. Acquired images were then analyzed using MATLAB to generate digital information such as plant greenness, plant body temperature, and plant quantum yield. Time series data of each spectral parameter show that plant greenness and chlorophyll fluorescence tend to decrease, while plant body temperature increase over time. However, distinctive spectral patterns were observed among the herbicides depending on their modes of action. Herbicide modes of action were then classified by machine learning of the parameters derived from plant image analysis, suggesting that plant image analysis can be used to diagnose not only herbicidal activity but also herbicide modes of action of unknown and novel herbicidal compounds.