A BIM-based virtual building system is developed by customizing Autodesk Revit to increase the efficiency of building rubble walls which is of high demand. The system consists of four modules for importing walls, processing and displaying stone images, virtual construction, and generating cutting diagrams and assembly diagrams respectively. In practical use, firstly, users should label numbers on stones, take photos of them and then import the photos into the system, and the system will generate the vector diagrams of each stone automatically. Secondly, based on the information of walls imported from BIM model and the vector diagrams, users construct walls virtually by means of interactive operations, such as movement, rotation and cutting, and the system will generate cutting diagrams of stones and assembly diagrams of walls at the same time which can be used to guide the construction. It is verified that the system could improve the efficiency of constructing rubble walls of high demand.