Non-contact methods have great potential as tools for assessing the health of structures because of their convenience in field implementation. In addition to wireless-based technologies, vision-based algorithms and low-cost cameras are an exciting new research area of the Structural Health Monitoring (SHM) field. They are expected to significantly advance future SHM systems and enable widespread adoption. In this study, a novel framework for structural identification using computer vision is proposed. Within this framework, the external loads (structural input) are estimated with video cameras and deep learning-based vehicle detection methods, while the structural responses (structural output) are monitored by tracking the motion of structural regions of interest with computer vision-based techniques. By synchronizing both input (loading monitored by cameras) and output (responses also captured by cameras) data of the structures, the structural system is identified using the following structural indices, Unit Influence Line (UIL), Unit Influence Surface (UIS), deflection profile and cross-correlation of displacements. These indices are further processed and utilized for damage detection and structural condition assessment of the structure. The proposed framework can improve the current SHM approaches by lowering the cost of testing and decreasing the testing time and required labour force. The results of the structure’s performance and health condition determined by the proposed framework are effective and satisfactory for use in decision making regarding the future management and maintenance of the structure.