Concrete defects pose a considerable threat to the safety and stability of concrete structures, so regular defect detection is essential for the maintenance of concrete structures. Compared with subjective and inefficient manual visual inspection, computer vision enjoys significant advantages in the automation of concrete defect detection, which has become a research hot spot in recent years. However, there is still a lack of comprehensive review in this field currently. Therefore, this paper aims to comprehensively analyze the research progress of computer vision technique in the field of concrete defect detection and classify the computer vision algorithms involved in concrete defect detection. Thereafter the paper summarizes the existing technical difficulties and analyzes the future research directions, which can provide a certain reference for the subsequent research in this field.