Computer Vision Model Training: A Step-by-Step Guide
You have inspection photos and know which defects you want a model to find. The difficulty is turning those photos into training data and knowing whether the model has learned enough to be useful. Training a computer vision model starts with labeled examples, followed by a training run and tests on images kept out of that run.
This guide follows one application of computer vision in building inspection: an object detector that marks suspected surface defects for an inspector to review. We’ll show the annotation files, dataset setup, and Python commands, then examine its predictions to find where it misses defects or flags the wrong features.
You’ll need to know how to run Python commands to follow along. By the end, you’ll know how to complete a training run and use the results to decide what needs further work.
1. Define the Visual Task and Required Annotations
Choose the task based on what the inspector needs to do with the result. If they’re sorting photographs for review, identifying which images contain cracks is enough. If they need to find individual cracks within a large façade photograph, the model must also mark their locations.
Take a photograph of a concrete façade with three separate cracks. The table below shows how the required annotations, or training labels, change with the task.
We’ll use object detection for this walkthrough because the inspector needs to locate suspected cracks within each photograph. For the three-crack example, that means preparing three labeled boxes. A single “crack present” label would leave out the locations the detector needs to learn.
Each box records a position in the image. For example, in a 4,000 × 3,000-pixel photograph, a box starting at (800, 600) and ending at (1,200, 1,500) encloses a region 400 pixels wide and 900 pixels high. These illustrative coordinates tell the training software where the labeled crack appears. They do not describe its physical width or length.
The resulting detector will flag suspected cracks for review. An inspector checks those findings; the model’s boxes do not establish the damage’s cause or severity.
2. Build and Check Your Inspection Dataset
Build the dataset around the photographs the detector will process during inspections. If those include wide façade views, training only on close-ups leaves out the smaller cracks and surrounding features it will encounter. Include both views, covering sunlit concrete and shaded areas beneath balconies.
Those photographs should also include surfaces without cracks. Construction joints, window edges and dark streaks resemble defects, so include reviewed examples where these features remain unmarked. Record the building and capture date as you collect the images; you’ll use those details to separate training photographs from those reserved for evaluation.
Agree on How to Label Difficult Images
Even with the right photographs, reviewers need to agree on what to mark. Ask two reviewers to independently annotate a small batch, such as 20 photographs, then compare their work before labeling the full collection.
Use disagreements to establish the annotation rules. If one reviewer puts a branching crack inside one box while another draws several boxes, examine the photograph together and save the agreed version as a reference. For this walkthrough, enclose visibly connected branches in one box and mark separate cracks individually.
Apply the same attention to partially hidden cracks. Where a pipe or ledge blocks the view, mark only the visible portion rather than guessing where the crack continues. If the feature itself is uncertain, hold the photograph for inspection review instead of assigning a doubtful label.
Once the rules are settled, use them to check the completed annotations for missed cracks and incorrectly marked joints or stains.
Keep Related Photographs in the Same Dataset Split
The next decision is which images the model learns from and which you reserve to assess its predictions. A collection of 1,000 photographs, for example, could be divided as follows. The counts illustrate a 70/15/15 split, not a required dataset size.
Make this split using the building records, rather than randomly distributing individual files. Two drone frames captured a second apart might show almost the same wall. If one enters training and the other enters testing, the test includes a view closely resembling something the model has already seen.
For an assessment on other buildings, keep each building’s photographs in one set, even if that changes the percentages above. Then check the range of conditions in each set. Holding back an entire building is useful, but if all its photographs are sunlit close-ups, they won’t test performance on shaded, wider views.
During development, use the validation set to compare revisions. Leave the test set untouched until the model has been selected. If you use its results to make further adjustments, you’ll need fresh held-back images for an independent assessment.
Check the Images the Model Will Actually Receive
After dividing the dataset, inspect the versions prepared for training. Resizing and cropping change what the model sees, so review these images beside the originals before starting the run.
For example, reducing a photograph from 4,000 pixels wide to 1,000 pixels wide reduces a feature’s width to one-quarter of its original size. A crack that appeared four pixels wide is reduced to roughly one pixel. At that scale, detail that distinguished it from a joint or stain might be lost.
Check the labels on these prepared images at the same time. Boxes must move with the marked features after cropping and scale with them after resizing. If a crack is no longer distinguishable or its box has shifted, fix the image preparation before training. The model needs both a visible example and a label in the right place.
3. Choose and Train Your Model
Start with a pretrained object detector and adapt it to the labeled inspection photographs. It has already learned patterns from other images; custom AI model training adjusts it to recognize the cracks you’ve marked. Its accuracy on those defects still needs testing.
Choose for the Inspection Workload
A detector flagging defects during capture must keep up with incoming images. Processing photographs after a drone survey allows a different deadline. Choose the starting model with that deadline and the available hardware in mind.
For example, processing 2,000 photographs within one hour gives the full job an average allowance of 1.8 seconds per photograph. Uploading, image preparation and saving predictions all count toward that hour. Measure the complete job on the planned hardware rather than relying on a supplier’s model-speed figure.
Meeting the deadline still leaves the question of detection quality. Incorrect flags add areas for inspectors to check, while missed cracks leave defects unflagged. Speed alone is insufficient grounds for selecting a model.
Set Up the First Training Run
Complete a short setup check before launching the full job: confirm that images and labels load, validation completes and a model version is saved. Once that succeeds, set the image size, batch size and training length.
With the earlier 700-image training set, 50 epochs means 50 passes through those examples, not 35,000 different photographs. More passes do not guarantee better detection. Early stopping ends training after a specified number of epochs without validation improvement, rather than continuing to the maximum regardless of results.
Record these settings with the starting model and dataset version. Later comparisons need to show whether a change came from the model, the photographs or the training setup.
Use Validation Results to Decide When to Stop
Falling training error shows that the model is improving on the examples it learns from. If it also starts missing more cracks or producing more incorrect flags on validation images, investigate overfitting. The improvement on familiar photographs is not carrying over to held-back ones.
Look for a sustained pattern across several checks rather than reacting to a single worse result. Keep saved versions and their validation results so an earlier version remains available if later training reduces detection quality. The final version saved is not automatically the best one to evaluate.

4. Evaluate Predictions and Improve the Model
Assess defect detection accuracy against limits set before evaluation: how many cracks the model is allowed to miss and how many incorrect flags inspectors will accept. Compare its predictions with inspector-checked annotations to establish whether it meets those requirements, and check whether it processes the photographs within the required time.
Read the Numbers Alongside the Photographs
Suppose the validation images contain 200 labeled cracks. The detector correctly locates 180, misses 20 and produces another 60 boxes around features that aren’t cracks.
For a detection to count as correctly located, its box must overlap the reference box by at least the amount specified in the evaluation. Keep that requirement unchanged across comparisons; lowering it would let less accurately placed boxes count as correct.
The counts also need a visual check. Review incorrect detections and missed cracks against the photographs, including images where the detector returned nothing. Checking only flagged areas leaves some misses unexamined.
Find What the Errors Have in Common
The overall score doesn’t show whether errors are concentrated in particular conditions. Finding 160 of 160 cracks in close-ups but only 20 of 40 in wider views still gives 90% recall. Yet the detector missed half the cracks in the wider photographs.
Break the results down by capture distance, material and lighting, then investigate the errors in each group:
- Misses concentrated in wider views: Compare the original photographs with the resized versions. If crack detail disappeared during resizing, test a larger input size before collecting more images.
- Construction joints repeatedly flagged: Check their training labels. Correct conflicting annotations, or add reviewed joint examples if they are missing.
Both cases involve poor detection, but they require different corrections. Adding more photographs won’t restore detail lost during image preparation or resolve contradictory labels.
Make One Change and Compare Again
If resizing caused the wider-view misses, test a larger input size while keeping the training photographs and other training settings unchanged. Compare the revised model on the same validation images using the same box-matching rule and confidence cutoff. The cutoff determines which predictions are retained based on their scores; changing it would also change the error counts.
Look for fewer wider-view misses without an increase in errors elsewhere. Then measure processing time again to establish whether the revised model still handles the survey within the required deadline.
Once the model and settings are selected, check them on the untouched test set. If those results lead to further tuning, use fresh held-back images for the next independent assessment. Photographs from conditions that still fail the requirements need manual review while the detector is improved.
Conclusion
You begin training a computer vision model by defining a clear task and gathering photos that illustrate what you want the model to recognize. Because the labels are consistent, it knows what features are defects. The training setup tells it how to process those examples. If you are losing crack detail on resize, or reviewers are labeling the same feature differently, these issues need to be fixed before it makes sense to do more training.
Testing reveals whether that preparation has produced a detector suited to the inspection work. Its performance on held-back photographs shows where it finds defects reliably and where misses or incorrect flags still need attention. Those results define the limits of its use, while inspectors verify the suspected defects it identifies.
Such verification requires access to the original imagery and the location of the defect. Our inspection intelligence platform, H3 Zoom, supports AI-assisted façade inspections by linking detected defects to their source images and positions on the building. Inspectors can review suspected cracks alongside that evidence and decide where a closer check is needed.
