Method for Developing Object and Image Recognition Systems Using Transfer Learning in Computer Vision

2.1 Data collection

A dataset of images is a structured collection of visual representations, categorized into one or more classes, stored locally or in the cloud, and designed to be processed and used for training machine learning (ML) models. In this phase, the image dataset [5, 6, 7, 8, 9] containing examples or data related to the task at hand is obtained. The framework begins with defining the task and designing the dataset to align it with existing resources, followed by data acquisition and immediate legal compliance verification. Subsequently, data quality is evaluated, and labeling adjustments are made, applying dataset augmentation if the information is insufficient. Finally, each stage is continuously documented and iterated as necessary, ensuring efficient, ethical, and effective use of available data and models while minimizing redundancy in the process. Data collection can be carried out through a logistical activity (e.g., setting up traffic cameras), a technical activity (software connected to a digital catalog database), or a commercial activity (purchasing an image archive) [10]. Data sharing and reuse [11]; discovery and exploration [12].

Task definition involves understanding the image recognition task to be performed; whether it is object recognition [13, 14, 15, 16] (locating multiple instances with bounding boxes), image classification [17, 18] (assigning a global label), or face detection [19, 20, 21, 22, 23, 24, 25] (identifying human faces, possibly extending to facial recognition). Each variant requires different technical approaches, from neural network architectures (YOLO (You Only Look Once), ResNet (Residual Network), MTCNN (multi-task cascaded convolutional neural network)) to specialized datasets (COCO (Common Objects in Context), ImageNet, WIDER Face), determining the choice of metrics and computational resources needed for effective implementation.

The design of the image dataset, or acquiring a representative dataset for the task, is a fundamental stage in developing computer vision systems. This process involves the careful collection of images from diverse sources, such as digital cameras, scanners, mobile devices, specialized sensors, or public databases [26] available in academic repositories and specialized platforms. Additionally, when existing data are insufficient or do not meet specific project requirements, it may be necessary to manually generate images [27] using techniques like controlled photography, three-dimensional (3D) rendering, or image synthesis via generative algorithms. It is crucial that the acquired dataset is diverse, balanced (SMOTE (Synthetic Minority Oversampling Technique)) [28], and representative [29, 30] of the real-world conditions where the system will operate, considering factors such as lighting variations, angles, resolutions, and possible noise or artifacts to ensure that trained models can generalize properly in practical scenarios.

Data acquisition is the process of finding datasets to train models [31]. Three approaches are identified [32]: Data discovery involves searching and indexing datasets in repositories such as corporate data lakes or the web using tools like DataHub, Google Fusion Tables, or Kaggle, which combine collaborative analysis, public management, and access to massive repositories. Search specializes in locating data either in corporate lakes or via HTML (Hypertext Markup Language) table extraction. Data augmentation complements existing datasets by integrating external sources or generating new data through crowdsourcing or synthetic data [31, 33]. Finally, data generation creates datasets from scratch using crowdsourcing (with procedural or declarative tasks) or synthetic techniques, such as generative adversarial networks (GANs) [34], reinforcement learning policies, or specific methods (Semantic Clustering and Partitioning (SCPN)/Semantic Extraction and Relation (SEAR) in natural language processing (NLP)) [32, 33].

Consent and legal compliance entail that collecting visual data involving people or objects in public or private spaces must be conducted under strict adherence to legal frameworks related to information protection and privacy [35, 36]. When collected data allow direct or indirect identification of individuals, obtaining informed, free, and explicit consent is essential, following principles of legality, transparency, and accountability. Complementary measures, such as data minimization [37, 38], anonymization [39, 40], and access controls, must be considered to ensure image processing is proportional, secure, and consistent with predefined purposes.

Ensuring the quality of collected data involves cleaning the data by removing defective images (artifacts, low resolution, distortions), correcting labeling errors by verifying that each label exactly corresponds to the visual content and fixing ambiguities or inconsistencies, and normalizing data [41] by standardizing formats, sizes, color spaces, and statistical distributions to guarantee consistency during training. Data quality [42, 43, 44] and labeling accuracy are critical, as errors in this phase can propagate during training and negatively impact the model’s final performance.

Labeling adjustments are made to effectively train a deep learning model, usually requiring prior data labeling [4, 45, 46]. This process assigns specific labels or categories [47] to each image in the dataset, enabling the model to learn patterns through supervised learning. Labeling data, when building knowledge bases from the web, initially assumes extracted facts are correct. Techniques include using existing labels [31]; crowd-based methods such as active learning and collective collaboration [32]; weak supervision, which generates imperfect labels at scale (e.g., data programming using multiple labeling functions instead of individual ones) [31, 48]; and data extraction leveraging existing knowledge bases to obtain entity attributes. These strategies balance volume, accuracy, and efficiency, adapting to contexts with initially scarce labels.

Dataset augmentation is performed if the existing image dataset is small and it is necessary to increase its size to improve model performance; this includes collecting more data, generating synthetic data, or increasing diversity within the dataset [2, 49, 50, 51].

Dataset documentation is very important and should include information about its origin, label structure, and any other relevant details so others can understand and use the dataset [52, 53].

2.2 Image preprocessing

This second phase consists of the manipulation and enhancement of images before they are used in model training or other applications.

Resizing involves adjusting the size of images while maintaining the aspect ratio (width/height). We can resize an image with a defined aspect ratio [54], which is the width of the image divided by its height.

Normalization ensures that pixel values are within a specific range, such as [0, 1] or [−1, 1] [55]. Normalization helps standardize data and speeds up the training process.

Adjusting the contrast and brightness of images improves the visibility of details [56]; this is useful when working with images of variable quality.

Cropping removes pixels from an image’s edges [57]; sometimes, it is necessary to crop the region of interest to focus on a specific part of the image for recognition tasks.

Rotation and transformation involve aligning all images to the same orientation or applying geometric transformations to correct distortions. In image processing, the rotation anchor point is the top-left corner at (0, 0) [58]. An image is rotated by an angle θ by transforming the (x, y) coordinates of each pixel relative to the origin [59, 60]. Morphological transformations, focused on the shape and structure of image features [61], include basic operations like erosion, which shrinks objects by removing edge pixels, and dilation, which expands objects by adding pixels [62]. Opening smooths contours and removes protrusions; closing fills holes and connects regions [63]. Advanced techniques such as hit-or-miss transformation (specific shape detection), morphological gradient (difference between dilation and erosion to highlight edges), and top-hat transforms (extracting small elements: white for bright, black for dark) optimize structure analysis [62, 64]. Algorithms like boundary extraction, hole filling, skeletonization (structural representation via iterative erosions), and convex hull (convexity approximation) allow describing shapes and patterns [65]. Operations such as thinning (thinning without losing connectivity) and pruning (removal of spurious pixels) refine results, while morphing (interpolation between images) gradually transforms structures [66]. Applicable to binary and grayscale images, these techniques are essential in pattern recognition, medical image processing, and computer vision [67]. Geometric transformations rearrange image pixels through basic operations, such as translation (displacement along x and y axes using matrices) [54, 57, 68], rotation (coordinate adjustment with trigonometric formulas for angle θ) [58, 60], scaling (resizing while preserving aspect ratio) [54, 69], flipping (reflection on axes) [58, 70], and cropping (extracting subregions using window and plop operators) [66]. Techniques like interpolation—nearest neighbor, bilinear, and bicubic—adjust resolution while preserving visual quality [71, 72], whereas advanced transformations like affine (combination of rotation, scaling, shear, and translation in homogeneous coordinates) [61] and projective (mapping quadrilaterals for perspective correction) [73] allow complex manipulation. Methods like deformation [61, 72], polar mapping [66], and barrel/pincushion transformations (non-linear radial distortion) [66] adapt images to specific geometric systems. Image arithmetic, consisting of addition, subtraction, and multiplication with values clipped to [0,255] [54, 68], and masking (focusing on specific areas) [54, 74] optimize data for analysis, while techniques like Procrustes alignment (translation, scaling, and rotation to preserve shape) [61] ensure accuracy in computer vision applications. These operations, based on matrix algebra and coordinate systems [60, 66], are essential in image processing to improve quality, adaptability, and computational efficiency.

Data augmentation creates modified images (rotations, shifts, brightness) to boost diversity and improve model generalization. Models trained with data augmentation on minority classes and transfer learning provide the highest classification accuracies [49]. There are many methods for data augmentation, among which Kumar [75] mentions deriving latent semantics, entity augmentation, and data integration. Advanced data augmentation techniques include adversarial training [76], synthetic data generation: GANs [34] or variational autoencoders (VAEs) [77], advanced geometric transformations, semantic-based augmentations, Mixup [78], CutMix [79], and controlled noise-based data augmentation.

Color correction involves adjusting the color balance of images to correct lighting and color representation problems [80, 81].

Image histogram transformation is an abstraction of an image where the frequency of each image value (contrast/brightness/intensity) is determined [59, 71, 74, 82]. A histogram shows the distribution of pixel intensities in an image, visualized as a graph that summarizes intensity levels [74]. Types of histograms include one-dimensional (1D) histograms or grayscale image histograms that graphically represent the distribution of pixel values, where the x-axis indicates intensities (0–255) and the y-axis their frequencies [74, 82]. In color images, histograms can analyze individual channels (RGB (Red, Green, Blue), HSL (Hue, Saturation, Lightness)) or two-dimensional (2D)/3D combinations, though the latter require reduced quantization to avoid complexity [64, 71]. Masks allow focusing on specific regions by calculating partial histograms to adjust contrast or segment objects [74]. Histogram equalization improves contrast by redistributing values through the cumulative distribution function (CDF), idealizing a uniform histogram [64, 83]. Techniques like histogram matching transform values to align distributions with a desired target, useful in color correction [83]. Additionally, histogram comparison and backprojection facilitate retrieving similar images or detecting objects through probability analysis [64, 71]. These tools are essential in digital processing to optimize quality, segmentation, and comparative analysis. Histogram transformation consists of aligning the histogram of an image to redistribute intensity levels to highlight certain features.

Regarding filtering and smoothing, noise can be caused by factors, such as poor lighting or distortion during image acquisition; filtering techniques are used to remove unwanted details, improve, smooth, or enhance features in images. Filters modify the appearance of an image through operations in the spatial domain (direct manipulation of pixels) or frequency domain (transforms like Fourier). In the spatial domain, linear filters (weighted combination of neighboring pixels) include the mean filter (smoothing but blurs edges) [61] and Gaussian filter (noise reduction with kernel based on σ) [67], while non-linear filters like the median filter (removes “salt and pepper” noise) [61] and order filters (max/min to enhance bright/dark points) [67] preserve edges. Edge detection filters use derivatives: Sobel and Prewitt (horizontal/vertical gradients) [84], Canny (non-maximum suppression and hysteresis thresholding) [85], and Laplacian (second derivative for zero crossings) [61]. In the frequency domain, low-pass filters (smoothing: ideal, Butterworth, Gaussian) [60] and high-pass filters (edge enhancement: Butterworth, Gaussian) [86] adjust spectral components, while techniques like CLAHE (contrast-limited adaptive histogram equalization) [87] and homomorphic filtering (illumination/reflectance correction via logarithm and Fourier) [88] enhance contrast in complex images. Advanced methods like Frangi filter (detection of tubular structures via Hessian eigenvalues) [89] and blur masking (sharpening by subtracting a smoothed version) [61] optimize applications in computer vision, medical processing, and image restoration.

Artifact removal involves eliminating unnecessary or irrelevant data from the image, such as watermarks, text, or unwanted artifacts.

Binarization consists of converting an image into a binary image where pixels are classified as black or white. This is useful for object segmentation applications.

Segmentation refers to separating objects of interest from the background or other objects in the image. Segmentation is useful when specific objects need to be identified and analyzed. It divides an image into homogeneous regions based on attributes such as discontinuities (edges) or similarities (intensity, texture) to facilitate interpretation [61, 62, 90]. Edge-based methods use operators like Sobel, Prewitt, Canny, and Scharr to identify abrupt intensity transitions [62, 64], while histogram-based approaches apply global thresholding (Otsu, variance maximization; Kapur, maximum entropy) or generalized entropies (Tsallis, Rényi) to separate objects and background [90, 91, 92]. In environments with variable lighting, adaptive thresholds (Niblack, Sauvola, Bernsen) dynamically adjust thresholds based on local statistics (mean, standard deviation, range), optimizing segmentation in subregions [82, 93, 94]. These techniques, fundamental in computer vision, enable applications such as Optical Character Recognition (OCR), medical diagnosis, and texture analysis, balancing accuracy and computational efficiency.

2.3 Pretraining

A pretrained model is a deep learning model trained on large datasets for general tasks like image or text recognition, ready to be adapted for various uses.

Regarding transfer learning, large pretrained models exhibit a remarkable ability to generalize and transfer knowledge across different datasets and tasks, making them highly effective tools for a wide range of applications. Transfer learning fine-tunes a pretrained model for a specific task or smaller dataset [76, 95]. Instead of training the model from scratch, the weights and parameters of the pretrained model are adjusted to adapt it to the new task. This is especially useful when the new dataset is small or when there is insufficient time to train a model from scratch, since the pretrained model has already captured useful general features. Transfer learning techniques can be implemented in various ways. One option is to use only the base architecture as a template, without incorporating the previously learned weights, which corresponds to the feature extraction approach. Another option is to leverage both the structure and pretrained weights of the original model, known as direct utilization. In this latter case, different strategies are possible: adjusting only certain layers, such as the output layer, to adapt them to the new domain; performing full fine-tuning by modifying all the model parameters during retraining; or expanding the architecture by adding new specialized layers at the end, which is associated with the multitask learning approach. Transfer learning methodologies are categorized into four approaches: feature extraction, fine-tuning, domain adaptation, and multitask learning [76], being a successful strategy in domains like computer vision and natural language processing [95].

Feature extraction uses a pretrained convolutional neural network’s (CNN’s) learned features as input for a new dataset or model without fine-tuning the entire model [16, 96].

In some cases, pretrained models are also used directly without fine-tuning, especially if the tasks are general ones for which the model was originally trained, such as object recognition in images or natural language processing. We can potentially improve the estimation of a local model by relying on the global model without using it directly; this idea can be called soft transfer learning, as opposed to hard transfer learning, where the global model is used directly without adjustments [97].

Fine-tuning focuses on adapting the upper layers while preserving the generic features learned in the initial layers from the source task, thereby reducing overfitting [76]. Fine-tuning a model trained on a source domain helps reduce the amount of data and training time required in a new target domain by avoiding training from scratch and decreasing data collection costs [95], significantly improving classification accuracy [96]. Fine-tuning adjusts a pretrained model’s weights for a task, improving performance but needing more computation. Feature extraction uses pretrained weights without changes, saving resources for small data [76, 96]; ablation studies help determine which layers should be fine-tuned depending on the case.

Domain adaptation reduces differences between source and target domains so models work well across them [95, 98]. Rule adaptation reuses general knowledge from previous tasks to facilitate new ones, while domain adaptation aligns datasets with similar distributions to enable the use of a single classifier across different contexts [96].

Multitask learning trains a model on multiple related tasks simultaneously, leading to better generalization and performance by sharing representations—particularly useful when data per task are limited [96]. However, it faces challenges, such as overfitting, catastrophic forgetting, and task interference; therefore, in NLP, separate models are often preferred for each task [99]. Multitask transfer learning improves performance and efficiency by sharing representations across related tasks in real-world applications.

2.4 Training

In this phase, a model is taught to recognize patterns, features, or classes in images.

The training dataset consists of labeled images, where each label indicates the class or category to which the image belongs. When training with reduced training datasets, Lu and Li [100] proposes a method combining data augmentation with transfer learning, allowing convolutional neural networks (CNNs) to avoid the overfitting problem and achieve excellent classification accuracy.

The choice of an appropriate model, such as a DCNN, is important. The model has layers of neurons that will be trained to learn patterns in images. Model selection involves balancing approximation and estimation errors, adding complexity due to the transfer distance between source and target distributions, which varies by hypothesis class [101].

Parameter initialization of the model with random values [102], which can be adjusted during training so that the model can make accurate predictions.

In optimization, a loss function measures the gap between model predictions and true labels in training.

The goal is to minimize loss by iteratively adjusting model parameters using optimization algorithms like gradient descent. In transfer learning, when pretrained models are reused and adapted to a new classification task, the cross-entropy loss function remains one of the most commonly used, especially in multi-class classification tasks. This function combines LogSoftmax (logarithm of the softmax function) and NLLLoss (negative-log-likelihood loss), just like in training from scratch [21].

The training process involves iterations through the training set. In each iteration, the model processes one or several images, makes predictions, and adjusts its parameters based on the calculated loss. This process repeats many times (epochs) until the model converges or a predefined stopping criterion is met.

When necessary, hyperparameters—learning rate, batch size, or architecture—can be adjusted [101].

2.5 Validation

It is performed after the training of the model and aims to evaluate its performance and generalization capability on unseen data.

The validation dataset is an independent set containing images not used during training; it adjusts hyperparameters to prevent overfitting. These images are also labeled with their corresponding categories, and the test set evaluates the final model performance.

Predictions on the validation set are made using the trained model. Each image is passed through the model, which generates class predictions or continuous values, depending on the specific task.

The analysis and interpretation of predictions aims to explain the model’s decisions, detect failures, and validate that it uses relevant features through interpretability tools. Some interpretability techniques include: Grad-CAM (Gradient-weighted Class Activation Mapping) [103], saliency maps, LIME (Local Interpretable Model-Agnostic Explanations) [104], SHAP (SHapley Additive Explanations) [105], feature importance in models, such as Random Forest or XGBoost, PDP (Partial Dependence Plot), and integrated gradients.

Performance evaluation consists of comparing the model’s output with the true labels in the validation set. The model’s performance is assessed using metrics appropriate for the task, such as accuracy, precision, recall, F1-score, among others [8], and these reflect the quality of the model’s predictions on unseen data.

Model hyperparameters may need adjustment if the performance on the validation set is unsatisfactory. Such adjustments can include changes in learning rate, model architecture, batch size, among others.

Validation helps detect and prevent overfitting, where the model fits training data too closely and fails on new data. Ahmed [1], Dawson et al. [7], and Alhanaee et al. [22] use image augmentation to expand training data, improving generalization and reducing overfitting. Selectively fine-tuning the top layers while sharing the initial layers with the source task reduces overfitting and favors better feature reuse [76].

In some cases, cross-validation is used, a technique that involves splitting the dataset into multiple partitions (folds) [1, 6, 7, 97] and performing multiple rounds of training and validation. This provides a more robust estimate of the model’s performance. Cross-validation allows evaluating the statistical significance between models, eliminating dependence on chance [6].

The model is saved once it achieves satisfactory performance, for later use in the recognition stage.

2.6 Recognition

It involves using a pretrained model to identify objects, patterns, or features in images, prerecorded videos, or real-time videos.

First, there must be a pretrained model; this model has learned to identify patterns and features in images, thanks to the training phase. The trained model can be used directly for the target task or a new task [98]. The model’s performance may degrade due to changes in the network, traffic patterns, usage behavior, or the incorporation of new devices [95].

The input data provide images for the model to recognize. These images can be captured in real-time through a camera or loaded from a file.

Before feeding the images to the model, preprocessing may be necessary, which includes normalization [55], resizing [54] and in some cases segmentation [61, 62, 90] or noise removal [60, 61, 67, 84, 85, 86, 87, 88].

The model processes the images and generates predictions; in most cases, it provides a class label or a probability associated with each class, depending on whether you are performing image classification or an object detection task.

Predictions may require post-processing depending on the application; this could include removing low-confidence predictions, grouping similar predictions, or generating additional information from the predictions, such as bounding boxes in the case of object detection.

The results are presented through the predictions in a meaningful way for the end user; this consists of displaying class labels, locations of detected objects in the image, or any other relevant information.

Automated decisions can be made or information provided based on the model’s predictions to a human user for informed decision-making.

Regarding iteration, the image recognition system can operate continuously, providing real-time predictions and improving over time as more data are collected and the model is recalibrated in real-world applications.