1. Bibliographic Information

1.1. Title

The central topic of the paper is "Rock Classification Based on Residual Networks".

1.2. Authors

The authors are Sining Zhoubian, Yuyang Wang, and Zhihuan Jiang, all affiliated with Tsinghua University. Their specific research backgrounds are not detailed in the abstract or introduction, but their work suggests expertise in computer vision and deep learning applications, particularly in the geological domain.

1.3. Journal/Conference

This paper was published on arXiv, a preprint server, on February 19, 2024. arXiv is widely used by researchers to rapidly disseminate their work before, or in parallel with, peer review and formal publication in journals or conference proceedings. While it allows for quick sharing of research, it means the paper might not yet have undergone formal peer review by a journal or conference.

1.4. Publication Year

1.5. Abstract

The paper addresses the challenge of rock classification, a critical geological problem, noting the insufficient exploration of this task using convolutional neural networks (CNNs). To overcome this, the authors propose two main approaches using residual neural networks. Firstly, they employ data augmentation to expand their limited dataset. By modifying a ResNet34 backbone—specifically, kernel sizes, normalization methods, and composition—they achieve an accuracy of 70.1% on the test dataset, a 3.5% improvement over a regular ResNet34. Secondly, they integrate multihead self-attention, similar to BoTNet, and introduce internal residual connections (IRC) within their model. This further boosts performance, reaching 73.7% accuracy on the test dataset. They also investigate the optimal number of bottleneck transformer blocks, finding that more than one block may not yield further performance improvements. The authors conclude by suggesting that their methodology and model design could inspire future research in rock classification and the development of new residual model architectures.

1.6. Original Source Link

The original source link is https://arxiv.org/abs/2402.11831, and the PDF link is https://arxiv.org/pdf/2402.11831v1.pdf. This indicates the paper is publicly available as a preprint on arXiv.

2. Executive Summary

2.1. Background & Motivation

The core problem the paper aims to solve is rock classification using computer vision methods. This problem is essential in geology as it provides important formation information.

The problem is important because relying solely on human visual identification for rock classification is time-consuming and requires significant expertise. Automating this process with computer vision can enhance efficiency and accessibility.

Specific challenges or gaps in prior research, as identified by the authors, include:

• Insufficient exploration: The application of convolutional neural networks (CNNs) to this problem has not been thoroughly investigated.

• Sub-optimal accuracy: Existing models constructed by other researchers for rock classification exhibit less-than-optimal accuracy.

• Data scarcity: The primary impediment to achieving higher accuracy is the insufficient amount of available data, with 53 rock varieties having "merely forty or fewer images for training" each. This limited data poses a significant challenge for training effective neural networks.

  The paper's entry point and innovative idea revolve around addressing these challenges by employing robust residual neural network architectures (ResNet34 and BoTNet-inspired models) combined with extensive data augmentation to compensate for data scarcity, and introducing novel architectural modifications like internal residual connections.

2.2. Main Contributions / Findings

The paper's primary contributions are:

• Effective Data Augmentation Strategy: Proposing and demonstrating the significant impact of data augmentation techniques (rotation, flipping, cropping, scaling, brightness, contrast, saturation, hue adjustments) on improving model accuracy for rock classification on small datasets.

• Optimized ResNet34 Architecture: Developing a modified ResNet34 by adapting principles from ConvNeXt, including replacing ReLU with GeLU, using fewer activation and normalization layers, substituting Batch Normalization with Layer Normalization, and incorporating 1x1 convolutional layers to form a Bottleneck structure. This optimized architecture achieved 70.1% accuracy.

• Novel Internal Residual Connection (IRC): Introducing an internal residual connection within bottleneck transformer blocks (which integrate multihead self-attention). This innovation is designed to make MHSA layers learn more effectively with less data and ensure model convergence, boosting performance to 73.7%.

• Analysis of Bottleneck Transformer Block Quantity: Exploring the influence of the number of bottleneck transformer blocks on model performance, finding that models with more than one block may not further improve performance and can even impair it without IRC.

• Application to Underexplored Field: Applying advanced deep learning techniques to the relatively underexplored field of geological image identification and classification, providing a foundation for future work.

  The key conclusions and findings reached are:

• Data augmentation is crucial for achieving high accuracy on small, specialized datasets.

• Architectural modifications (inspired by ConvNeXt and BoTNet) to ResNet34 can significantly enhance performance.

• Integrating multihead self-attention is beneficial, but its effective application, especially with limited data, can be improved by adding internal residual connections.

• More complex models (e.g., more bottleneck transformer blocks) are not always better, especially when data is scarce or without appropriate architectural safeguards like IRC.

• The proposed approaches offer a promising direction for improving automated rock classification.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand this paper, a reader should be familiar with the following fundamental concepts in deep learning and computer vision:

• Convolutional Neural Networks (CNNs):

  • Conceptual Definition: A class of deep neural networks, most commonly applied to analyzing visual imagery. CNNs are designed to automatically and adaptively learn spatial hierarchies of features from input data, typically images. They achieve this through specialized layers that perform convolutions.
  • Key Components:
    • Convolutional Layers: These layers apply a set of learnable filters (kernels) to the input image, performing a convolution operation to create feature maps. Each filter specializes in detecting a specific feature (e.g., edges, textures).
    • Pooling Layers: These layers reduce the spatial dimensions (width and height) of the feature maps, thereby reducing the number of parameters and computation in the network. Common types include max pooling and average pooling. This also helps in achieving translational invariance.
    • Activation Functions: Non-linear functions applied to the output of neurons to introduce non-linearity into the model, allowing it to learn complex patterns. Common examples include ReLU (Rectified Linear Unit) and GeLU (Gaussian Error Linear Unit).
    • Fully Connected Layers: Traditional neural network layers where every input is connected to every output. These are typically used at the end of a CNN to perform classification based on the features extracted by the convolutional layers.

• Residual Networks (ResNets):

  • Conceptual Definition: A type of CNN architecture that addresses the problem of vanishing/exploding gradients and degradation (accuracy saturation and then degradation) in very deep networks. They achieve this by introducing "skip connections" or "residual connections".
  • Skip Connection: A direct connection that bypasses one or more layers. The output of such a block is $F(x) + x$, where $x$ is the input to the block and F(x) is the output of the stacked layers. Instead of learning the mapping H(x), the block learns the residual mapping $F(x) = H(x) - x$. It is often easier to optimize the residual mapping than to optimize the original, unreferenced mapping.
  • Vanishing Gradient Problem: In very deep neural networks, gradients can become extremely small during backpropagation, effectively preventing the network from learning. Skip connections help by providing an alternative path for gradients to flow.
  • ResNet34: A specific architecture in the ResNet family, consisting of 34 layers. It uses BasicBlocks (two 3x3 convolutional layers with a skip connection) and global average pooling.

• Data Augmentation:

  • Conceptual Definition: A set of techniques used to increase the amount of data by adding slightly modified copies of already existing data or newly created synthetic data from existing data. It helps to prevent overfitting and improve the generalization ability of the model, especially when the original dataset is small.
  • Common Techniques: Examples include rotation, flipping (horizontal/vertical), cropping, scaling, translation, changes in brightness, contrast, saturation, and hue.

• Activation Functions (ReLU, GeLU):

  • ReLU (Rectified Linear Unit): A popular activation function defined as $f(x) = \max(0, x)$. It outputs the input directly if it's positive, otherwise, it outputs zero.
  • GeLU (Gaussian Error Linear Unit): A smoother, more recent activation function defined as $f(x) = x \cdot \Phi(x)$, where $\Phi(x)$ is the cumulative distribution function for the standard Gaussian distribution. It is known to perform well in transformer models and can mitigate gradient vanishing better than ReLU due to its smoothness.

• Normalization Layers (Batch Normalization, Layer Normalization):

  • Conceptual Definition: Techniques used to normalize the activations of internal layers of a neural network. This helps in stabilizing and accelerating the training process by reducing the internal covariate shift (the change in the distribution of network activations due to the change in network parameters during training).
  • Batch Normalization (BN): Normalizes activations across the batch dimension. For each feature, it computes the mean and variance over the mini-batch and normalizes the values.
  • Layer Normalization (LN): Normalizes activations across the feature dimension for each individual sample, independently of other samples in the mini-batch. This is often preferred in models like Transformers where batch size can vary or be small, or when sequence length varies.

• Bottleneck Blocks:

  • Conceptual Definition: A type of residual block used in deeper ResNets (like ResNet50, 101, 152) to reduce computational cost. Instead of two 3x3 convolutional layers, it uses a sequence of 1x1, 3x3, and 1x1 convolutional layers. The initial 1x1 convolution reduces the dimensionality, the 3x3 convolution processes the reduced representation, and the final 1x1 convolution expands the dimensionality back to the original. This "bottleneck" structure is more computationally efficient for deeper networks.

• Multi-head Self-Attention (MHSA):

  • Conceptual Definition: A core component of Transformer models. It allows the model to weigh the importance of different parts of the input sequence (or different regions in an image) when processing a particular element.
  • Self-Attention: An attention mechanism where the input sequence attends to itself. For each element in the input, it computes a weighted sum of all other elements, where the weights are determined by the similarity between the query element and other key elements. This allows the model to capture dependencies between distant elements.
  • Multi-head: Instead of performing a single attention function, MHSA performs multiple attention functions (heads) in parallel. Each head learns to focus on different parts of the input or different types of relationships. The outputs from these multiple heads are then concatenated and linearly transformed to produce the final output.
  • Role in Vision: While initially dominant in NLP, MHSA has been adapted for computer vision (e.g., in Vision Transformers, BoTNet) to capture global dependencies in images that CNNs might miss due to their local receptive fields.

3.2. Previous Works

The paper builds upon several established and recent deep learning architectures:

• ResNets (Residual Networks):

  • Developed by Kaiming He et al. (ResNet34 is specifically mentioned).
  • Core innovation: Introduced residual learning and skip connections to enable the training of very deep neural networks by mitigating the vanishing/exploding gradient problem and the degradation problem.
  • Architecture: ResNet34 consists of 34 layers, utilizing residual blocks that learn residual functions. It uses skip connections to add the input of a block to its output before activation.
  • Impact: Revolutionized deep learning, allowing for much deeper and more performant models in image classification, object detection, and semantic segmentation. The paper uses ResNet34 as its foundational model due to its effectiveness, ease of implementation, and wide adoption.

• ConvNeXt:

  • Proposed by Liu Zhuang et al. [1].
  • Context: Inspired by the success of Swin Transformer in vision tasks, ConvNeXt systematically re-evaluated and modernized traditional convolutional networks. The goal was to demonstrate that purely ConvNet models could achieve competitive performance with Transformers while maintaining faster inference speeds.
  • Key modifications (starting from ResNet-50 or ResNet-200):
    • Macro design: ResNet-like stages but with fewer channels in early stages.
    • Depth-wise separable convolution: Using depthwise convolutions and pointwise convolutions to reduce parameters and computation.
    • Inverted bottleneck layer: Similar to MobileNetV2, where channels are expanded before a depthwise convolution and then projected back.
    • Large convolutional kernel: Using larger kernel sizes (e.g., 7x7) in depthwise convolutions to capture larger receptive fields.
    • Detailed design optimizations: Replacing ReLU with GeLU, using fewer activation functions and normalization layers, substituting Batch Normalization with Layer Normalization, and decomposing downsampling layers.
  • Impact: Showed that ConvNets could be competitive with Transformers by adopting some of their design principles, enhancing the potential of traditional convolutional networks. The current paper draws inspiration from ConvNeXt for its kernel modifications.

• BoTNet (Bottleneck Transformers for Visual Recognition):

  • Proposed by Aravind Srinivas et al. [2].
  • Context: A hybrid architecture that combines convolutional networks with Transformer's multi-head self-attention (MHSA) mechanism.
  • Core idea: Replaces one of the convolutional layers in ResNet's last residual connection block with an MHSA layer. This aims to leverage MHSA's ability to capture global dependencies while retaining the efficiency of CNNs for local features.
  • Experiments: Explored replacing convolutional layers at different locations in a ResNet50 backbone. Found that replacing the last convolution layer achieved the highest test accuracy.
  • Performance: BoTNet50 showed an improvement of about 1% to 5% over ResNet50 on some image classification tasks.
  • Impact: Demonstrated the effectiveness of integrating MHSA into ConvNet backbones for visual recognition, inspiring the current paper's use of MHSA and bottleneck transformer blocks.

3.3. Technological Evolution

The evolution of technology relevant to this paper can be traced from traditional machine learning to deep learning, and then within deep learning, from basic CNNs to advanced architectures.

1. Traditional Machine Learning (e.g., SVM, Random Forest): Early approaches to image classification often relied on hand-engineered features combined with simpler machine learning models. The paper notes these models "may prove too simplistic for such a complex task," highlighting their limitations in feature extraction for complex visual data.

2. Early Convolutional Neural Networks (CNNs): These marked a significant leap by automatically learning features from raw image data. However, as networks grew deeper, they faced challenges like vanishing gradients and degradation.

3. Residual Networks (ResNets): The introduction of ResNets revolutionized deep learning by addressing the training difficulties of very deep networks through residual connections. This allowed for the construction of much deeper and more powerful CNNs, becoming a cornerstone for many vision tasks. ResNet34 is a prime example of this generation.

4. Modern CNN Architectures (e.g., ConvNeXt): While Transformers gained prominence, ConvNeXt demonstrated that CNNs could still be highly competitive by systematically adopting modern design principles (like large kernels, inverted bottlenecks, Layer Normalization, and specific activation functions) often found in Transformers. This showed that CNNs still had significant untapped potential.

5. Hybrid Architectures (e.g., BoTNet): Recognizing the strengths of both CNNs (local feature extraction, inductive biases) and Transformers (global context modeling via attention), hybrid models like BoTNet emerged. These models aim to combine the best of both worlds, using CNNs for initial feature extraction and then integrating attention mechanisms for richer contextual understanding.

   This paper fits into this timeline by taking ResNet34 as a foundation, applying ConvNeXt-inspired modifications to enhance its ConvNet capabilities, and then further integrating BoTNet-inspired multihead self-attention to capture global information. Crucially, it introduces internal residual connections to specifically address challenges faced by MHSA in data-scarce environments, pushing the boundaries of hybrid ConvNet-Transformer architectures.

3.4. Differentiation Analysis

Compared to the main methods in related work, the core differences and innovations of this paper's approach are:

• Target Application & Data Scarcity Focus: Unlike ConvNeXt and BoTNet which were evaluated on large-scale datasets like ImageNet, this paper specifically targets rock classification, a domain characterized by extremely limited data (e.g., "forty or fewer images for training" per class). This context drives the necessity for robust data augmentation and efficient model designs.

• Comprehensive ResNet34 Modification:

  • While ConvNeXt provided general guidelines for modernizing ConvNets, this paper applies and evaluates specific ConvNeXt-inspired modifications (e.g., GeLU, Layer Normalization, Bottleneck structure for ResNet34's BasicBlock) within the context of rock classification. This is a specific application and validation of these principles.

• Novel Internal Residual Connection (IRC):

  • BoTNet showed the benefit of incorporating MHSA into ResNets. However, the current paper's most significant architectural innovation is the introduction of internal residual connections within the bottleneck transformer block. This is a direct response to the challenge of MHSA requiring large datasets and potential convergence issues when combined with ConvNets. The IRC allows the MHSA layer to learn only the "second-order difference," simplifying its learning task and making it more robust for smaller datasets. This is a novel adaptation to make attention mechanisms more data-efficient.

• Systematic Exploration of Hybrid Components: The paper systematically investigates:

  • The impact of ConvNeXt-inspired kernel modifications.
  • The effect of multihead self-attention (via bottleneck transformer blocks).
  • The optimal number of bottleneck transformer blocks.
  • The crucial role of the proposed internal residual connections. This multi-faceted exploration for the specific rock classification task differentiates its experimental focus.

• Combined Approach (Proposed for Future Work): The paper explicitly points out that combining the kernel modification (from ConvNeXt inspiration) with the bottleneck transformer block and internal residual connections has not yet been explored but is a promising future direction. This suggests the current work lays the groundwork by validating each component's effectiveness independently or in limited combinations.

  In essence, while the paper draws heavily from ResNet, ConvNeXt, and BoTNet, its unique contribution lies in applying and adapting these advanced architectures to a challenging, data-scarce domain (rock classification) through a novel internal residual connection design, and systematically validating the efficacy of these modifications.

4. Methodology

4.1. Principles

The core idea of the method used is to leverage advanced residual network architectures and data augmentation techniques to improve the accuracy of rock classification, particularly in scenarios with limited training data. The theoretical basis and intuition behind this approach are:

1. Overcoming Data Scarcity: Deep neural networks typically require large datasets for effective training. For specialized tasks like rock classification where data is naturally limited, data augmentation is crucial. The principle is to artificially increase the diversity and quantity of training data by applying various transformations to existing images, thereby making the model more robust and improving its generalization ability.
2. Enhancing Feature Extraction with Residual Learning: ResNets are chosen as the backbone because they address the vanishing gradient problem, allowing for the training of deeper networks that can learn more complex and hierarchical features from images. This is fundamental for capturing the subtle visual cues needed for accurate rock classification.
3. Modernizing ConvNet Architectures: Drawing inspiration from ConvNeXt, the paper aims to update traditional ResNet components by incorporating modern ConvNet design principles. This includes using smoother activation functions (GeLU), more robust normalization methods (Layer Normalization), and bottleneck structures to improve computational efficiency and model performance.
4. Integrating Global Context with Attention: The BoTNet concept is adopted to integrate multihead self-attention (MHSA) into the convolutional backbone. The principle here is that while CNNs are excellent at local feature extraction, MHSA can capture long-range dependencies and global contextual information, which might be crucial for distinguishing between rock types with similar local textures but different overall patterns.
5. Stabilizing Hybrid Models with Internal Residual Connections: Acknowledging that MHSA layers often require large amounts of data and can slow down convergence or even prevent it in hybrid models, the internal residual connection (IRC) is introduced. The intuition is that by providing a skip connection directly across the MHSA layer, the attention mechanism only needs to learn the "difference" or "residual" transformation, making its learning task simpler, more stable, and less data-intensive. This is a critical principle for making attention practical in data-constrained scenarios.

4.2. Core Methodology In-depth (Layer by Layer)

The paper proposes two main approaches: kernel modification based on ConvNeXt and Bottleneck Transformer with Internal Residual Connections based on BoTNet, both built upon a ResNet34 backbone and augmented data.

4.2.1. Data Augmentation

Upon acquiring the dataset, a comprehensive strategy of data augmentation is employed to enlarge the dataset and enhance the model's ability to generalize.

The data augmentation process involves several transformations:

• Rotation: Changing the orientation of the image.

• Horizontal and Vertical Flipping: Mirroring the image along its horizontal or vertical axis.

• Cropping: Selecting a part of the image.

• Scaling: Resizing the image.

• Adjustments to Image Attributes: Modifying brightness, contrast, saturation, and hue.

  These augmentations are chosen to simulate the natural variability in rock formations and environmental conditions.

The following figure (Figure 1 from the original paper) shows the data augmentation process:

Figure 1. The data augmentation process

The process introduces variations in orientation, geological features, lighting conditions, and other environmental factors, making the model more robust.

The following figure (Figure 2 from the original paper) shows some examples randomly extracted after data augmentation:

Figure 2. Some examples randomly extracted after data augmentation

4.2.2. Kernel Modification

This approach is inspired by ConvNeXt's optimization strategies and is applied to the ResNet34 backbone. The modifications aim to improve the internal details within the residual blocks.

The specific modifications are:

• Activation Function Replacement: The ReLU function is replaced with the GeLU function. The GeLU function is generally smoother than ReLU, which can help in mitigating the vanishing gradient problem to some extent and has shown better performance in modern architectures.

• Fewer Activation Functions and Normalization Layers: The network uses fewer activation functions and normalization layers. This strategy is inspired by Transformer principles, which often feature simpler architectures in terms of these components compared to traditional CNNs.

• Normalization Layer Substitution: Batch Normalization (BN) layers are replaced with Layer Normalization (LN) layers. Layer Normalization normalizes features across the channels for each sample independently, which can be more stable than Batch Normalization (which normalizes across the batch) for varying batch sizes or when combined with attention mechanisms.

• Architectural Change to Bottleneck Structure: The original BasicBlock in ResNet34 is replaced with a BottleNeck structure. This involves adding a 1x1 convolutional layer at the beginning of each residual block. The purpose of this 1x1 convolution is to reduce the number of channels before the main 3x3 convolution and then expand them back, which helps in reducing computation and parameter count, thus improving the model's performance and computational speed.

  The following figure (Figure 4 from the original paper) shows the structure of the modified kernel:

  Figure 4. The structure of the modified kernel.

The modified block consists of a 1x1 convolution, followed by a 3x3 convolution, with both having Layer Normalization (LN) and GeLU activation functions, and a residual connection adding the input features to the output of these layers.

4.2.3. Bottleneck Transformer

This approach incorporates multihead self-attention (MHSA) into the CNN model, drawing inspiration from BoTNet.

• Target Layers: The MHSA layers are specifically introduced into the last two bottleneck blocks of ResNet34. These are chosen as the main target for modification.
• Block Formation: An MHSA layer is inserted into the two 3x3 kernel convolution layers (which typically include batch normalization and ReLU) to form a bottleneck transformer block. This replaces the standard convolutional layers in these specific ResNet34 blocks with an attention mechanism. The goal is for the MHSA to capture global dependencies that CNNs might miss.

4.2.4. Internal Residual Connections (IRC)

The internal residual connection (IRC) layout is designed to address challenges associated with MHSA layers, particularly their requirement for large quantities of training data and potential convergence issues in hybrid convolution-attention models.

• Design: While maintaining the existing convolutional layers, MHSA layers, and the overall residual connection outside the bottleneck transformer block, an additional residual connection is added across the MHSA layer itself.

• Purpose: This IRC means that the MHSA layer no longer needs to learn the entire complex mapping from its input to output. Instead, it only needs to learn the "second order difference" of the data. This simplifies the MHSA's task, making it easier to train, requiring less data, and enabling it to learn more subtle patterns effectively. It also helps in improving the convergence stability of models that combine convolution and attention.

  The following figure (Figure 3 from the original paper) demonstrates the layouts of a common bottleneck transformer block and one with IRC:

  Figure 3. Layout of a bottleneck transformer block. Left: A normal bottleneck transformer block. Right: A bottleneck transformer block with internal residual connection.

• Left (Normal Bottleneck Transformer Block): Shows an input going through a 1x1 Conv, then BN, ReLU, then MHSA, then another BN, ReLU, and finally a 1x1 Conv. The output of this sequence is added to the original input via an outer residual connection.

• Right (Bottleneck Transformer Block with Internal Residual Connection): Shows the same structure as the left, but critically, there is an additional skip connection that adds the input of the MHSA layer directly to its output. This creates the internal residual connection, making the MHSA layer learn the residual.

4.2.5. Model Architectures

The paper investigates different configurations of the ResNet34 backbone, specifically focusing on the last layer's structure.

The following figure (Figure 5 from the original paper) illustrates the layouts of the last convolution layer:

Figure 5. Layouts of the last convolution layer. Left: Original ResNet34 layout. Middle: One bottleneck transformer block. Right: Two bottleneck transformer blocks.

• Left (Original ResNet34 layout): Depicts the standard ResNet34 last layer, typically composed of multiple convolutional blocks.
• Middle (One bottleneck transformer block): Shows the integration of a single bottleneck transformer block (which may include IRC) into the last layer of ResNet34.
• Right (Two bottleneck transformer blocks): Illustrates a more complex configuration where two bottleneck transformer blocks are used in sequence within the last layer.

5. Experimental Setup

5.1. Datasets

The experiments were conducted on a dataset for rock classification.

• Source: Not explicitly mentioned, but it's for geological image identification.
• Scale: The dataset is described as insufficient and small. It comprises 53 distinct rock varieties. For each variety, there are "merely forty or fewer images for training." This extreme data scarcity is a core challenge the paper aims to address.
• Characteristics: The images are visual representations of various rock types. Given the context, they likely feature different textures, colors, and patterns characteristic of geological samples.
• Domain: Geology and Earth sciences, specifically within petrography or geological surveying.
• Why these datasets were chosen: This specific dataset represents a real-world challenge in geology where data collection can be difficult and expensive. It is highly effective for validating methods designed to perform well under data-constrained conditions, such as those employing data augmentation and robust model architectures. The paper doesn't provide a concrete example of a data sample beyond referring to "rock samples" and "images", but Figure 2 visually presents some examples after augmentation.

5.2. Evaluation Metrics

The primary evaluation metric used in the paper is Accuracy.

• Conceptual Definition: Accuracy is a common metric for classification tasks, representing the proportion of total predictions that were correct. It measures how well the model predicts the true class of each instance.
• Mathematical Formula: $ \text{Accuracy} = \frac{\text{Number of Correct Predictions}}{\text{Total Number of Predictions}} $
• Symbol Explanation:
  • Number of Correct Predictions: The count of instances where the model's predicted class label matches the actual (ground truth) class label.
  • Total Number of Predictions: The total number of instances (e.g., images) in the dataset being evaluated.

5.3. Baselines

The paper's method is compared against several configurations, primarily variations of ResNet34 and BoTNet-inspired models, rather than external, distinct baseline models.

The baseline configurations include:

• ResNet-34 on initial dataset: This is the absolute baseline, representing the performance of a standard ResNet34 without any modifications or data augmentation on the raw, limited dataset.
• ResNet-34 with data augmentation: This serves as a baseline to demonstrate the impact of data augmentation alone, showing the improvement before any architectural modifications.
• Modified ResNet-34 (with kernel modifications): This configuration compares the ResNet34 modified with ConvNeXt-inspired principles (e.g., GeLU, Layer Normalization, Bottleneck structure) against the standard ResNet34 (with data augmentation).
• BoTNet-like structures:

  • Models with varying numbers of bottleneck transformer blocks (0, 1, 2) without internal residual connections. The "0 BoT" case essentially corresponds to a ResNet34 (with data augmentation and potentially kernel modifications, though not explicitly combined in the tables for direct comparison).

  • Models with internal residual connections applied to bottleneck transformer blocks (1 or 2 blocks).

    These baselines are representative because they systematically isolate and evaluate the impact of the proposed changes (data augmentation, kernel modifications, MHSA integration, and IRC) starting from a widely recognized and fundamental CNN architecture like ResNet34.

6. Results & Analysis

6.1. Core Results Analysis

The experimental results strongly validate the effectiveness of the proposed methods, particularly data augmentation and the introduction of internal residual connections, in improving rock classification accuracy on a small dataset.

• Impact of Data Augmentation: The initial performance of a plain ResNet34 on the raw dataset was very low at 30.64%. Applying data augmentation alone drastically improved this to 66.64%, highlighting its critical role in mitigating the effects of data scarcity. This shows that for specialized tasks with limited data, data augmentation is a fundamental first step.
• Effectiveness of Kernel Modifications: Further modifications to the ResNet34 kernel, inspired by ConvNeXt (such as GeLU, Layer Normalization, and 1x1 convolutional layers for Bottleneck structure), consistently improved accuracy. Starting from 66.6% (baseline with data augmentation), these modifications incrementally led to 70.1% accuracy. This confirms that modernizing ConvNet components can yield substantial gains.
• Bottleneck Transformer and IRC Synergy:

  • The bottleneck transformer with one block showed a modest improvement (67.0% vs 66.6% for common models). However, adding a second bottleneck transformer block without IRC actually impaired performance (65.9%), suggesting that more complexity isn't always better, especially with limited data.

  • The internal residual connection (IRC) proved highly effective. When applied to a model with one bottleneck transformer block, it achieved the highest accuracy of 73.7%. This is a significant jump compared to the common model with one BoT block (67.0%) and the modified ResNet34 (70.1%).

  • Even with two BoT blocks, IRC improved performance (70.7% with IRC vs 65.9% without IRC), though it did not surpass the performance of one BoT block with IRC. This reinforces the idea that IRC helps stabilize and improve MHSA performance, particularly when the network becomes more complex or when data is scarce.

    In summary, the results demonstrate a progressive improvement strategy: data augmentation first, then ConvNeXt-inspired ConvNet optimizations, and finally, BoTNet-inspired attention mechanisms augmented with the novel internal residual connection for optimal performance. The findings also provide practical guidance that integrating MHSA should be done judiciously, and IRC is key to making MHSA effective in data-constrained scenarios.

6.2. Data Presentation (Tables)

The following are the results from Table 1 of the original paper:

The following are the results from Table 2 of the original paper:

The following are the results from Table 3 of the original paper:

6.3. Ablation Studies / Parameter Analysis

The paper conducts several experiments that can be interpreted as ablation studies and parameter analyses to evaluate the effectiveness of different components and hyper-parameters.

• Ablation Study on Data Augmentation:

  • Table 1 clearly shows an ablation on the effect of data augmentation.
  • ResNet-34 on initial dataset: 30.64% accuracy.
  • ResNet-34 with data augmentation: 66.64% accuracy.
  • Result: Data augmentation alone provides a massive 35.00% absolute increase in accuracy, demonstrating its foundational importance for this task.

• Ablation Study on Kernel Modifications (Inspired by ConvNeXt):

  • Table 2 presents a sequential ablation-like study on the individual effects of kernel modifications building on the ResNet-34 with data augmentation baseline (which achieved 66.6% accuracy, represented as BasicBlock in Table 2).
  • BasicBlock (ResNet-34 with data augmentation): 66.6%
  • Replace Relu with Gelu: No change, 66.6% (suggests GeLU alone might not yield a direct large increase, or its benefits are more pronounced in combination).
  • Fewer activation functions: 67.8% (an increase of 1.2%).
  • Replace BN with LN: 68.3% (an increase of 0.5%).
  • Add a 1x1 convolutional layer (for Bottleneck structure): 70.1% (a significant increase of 1.8%).
  • Result: These modifications, particularly using fewer activation functions, replacing BN with LN, and adopting a Bottleneck structure, cumulatively enhance the model's performance, leading to a 3.5% total increase from the augmented ResNet-34 baseline.

• Parameter Analysis on Number of Bottleneck Transformer Blocks and Ablation on Internal Residual Connection (IRC):

  • Table 3 directly compares models with 0, 1, and 2 bottleneck transformer (BoT) blocks, with and without IRC. All models implicitly include data augmentation. The "0 BoT" common model also includes the kernel modifications that achieved 70.1%, but the table starts with 66.6% for "0 BoT" common, suggesting it might be the ResNet34 with only data augmentation and without kernel modifications. For consistency with the abstract's 70.1% for modified ResNet34, there's a slight ambiguity, but the trends are clear.
  • Common (without IRC):
    • 0 BoT: 66.6% (This is likely the ResNet34 with data augmentation).
    • 1 BoT: 67.0% (Slight improvement of 0.4% by adding one BoT block).
    • 2 BoT: 65.9% (Performance drops by 1.1% when adding a second BoT block, indicating diminishing returns or even degradation without IRC).
  • Use IRC:
    • 0 BoT: Not applicable/tested for IRC as IRC is within BoT blocks.
    • 1 BoT: 73.7% (Massive increase of 6.7% over 1 BoT common, and 7.1% over 0 BoT common). This is the highest accuracy achieved.
    • 2 BoT: 70.7% (An increase of 4.8% over 2 BoT common, but still lower than 1 BoT with IRC).
  • Result: IRC consistently improves the model's performance, especially when bottleneck transformer blocks are present. The optimal configuration for BoT blocks with IRC is one block, achieving 73.7%. Adding more BoT blocks beyond one, even with IRC, does not further improve performance and can lead to a slight decrease compared to the single BoT IRC setup. This indicates a sweet spot for complexity when integrating attention into ConvNets for data-limited scenarios.

7. Conclusion & Reflections

7.1. Conclusion Summary

The paper successfully demonstrates that deep learning, specifically leveraging residual networks and their modern adaptations, can effectively address the challenging problem of rock classification, particularly when faced with limited data. The key findings are:

• Data augmentation is paramount: It significantly boosts model accuracy on small datasets, transforming an initially poor performance (30.64%) into a respectable one (66.64%).
• ConvNeXt-inspired kernel modifications are effective: Adapting elements like GeLU, Layer Normalization, and Bottleneck structures to ResNet34 further improves accuracy (up to 70.1%).
• Bottleneck Transformer blocks can enhance performance: Integrating multihead self-attention via one BoT block showed a slight improvement. However, merely increasing the number of these blocks without further modifications can be detrimental.
• Internal Residual Connection (IRC) is a critical innovation: This novel mechanism within the bottleneck transformer block stabilizes MHSA learning, making it more robust and effective in data-scarce environments. It led to the highest accuracy (73.7%) with one BoT block.
• The work underscores the potential of advanced residual network designs for underexplored geological imaging tasks.

7.2. Limitations & Future Work

The authors acknowledge the following limitations and propose future research directions:

• Insufficient Datasets: The most significant limitation is the small size of existing datasets for rock classification. Future work should prioritize acquiring new data and expanding these datasets to enable more robust model training and potentially unlock higher performance ceilings.
• Unexplored Combinations: The paper notes that it did not explore combining both the kernel modification approach (inspired by ConvNeXt) and the bottleneck transformer with internal residual connections. This suggests that the highest reported accuracy (73.7%) might be further improved by integrating these two successful strategies. This is highlighted as a "prospective approach for a wider range of tasks."

7.3. Personal Insights & Critique

This paper offers valuable insights into tackling specialized image classification tasks with limited data, a common real-world constraint.

• Inspirations:

  • The systematic approach to addressing data scarcity, starting with comprehensive data augmentation and then iteratively refining the model architecture, is a strong methodological takeaway.
  • The internal residual connection is a clever and practical architectural innovation. It highlights that simply adopting complex Transformer components isn't enough; thoughtful modifications are needed to adapt them to specific data conditions. This principle could be transferable to other domains where hybrid CNN-Transformer models struggle with limited data or convergence.
  • The empirical finding that "more is not always better" (regarding the number of bottleneck transformer blocks without IRC) is a crucial reminder for model design, emphasizing the importance of balancing complexity with data availability and architectural stability.

• Potential Issues, Unverified Assumptions, or Areas for Improvement:

  • Dataset Details: While the paper mentions 53 rock varieties with limited images, it doesn't provide specific information about the dataset's source, characteristics (e.g., image resolution, typical visual challenges), or public availability. This makes reproducibility and direct comparison with future work challenging. A more detailed dataset description, perhaps including sample images of each class, would enhance the paper's impact.

  • Vagueness of "Similar Backbone like BoTNet": The paper states using a "similar backbone like BoTNet" but doesn't fully detail how the BoTNet architecture was adapted beyond the integration of MHSA into ResNet34's last layers. Explicitly outlining the full BoTNet-inspired architecture used would improve clarity.

  • Lack of Baselines from Other SOTA Models: The comparison is primarily against variations of ResNet34 and BoTNet-inspired designs. While this allows for clear ablation, comparing against other state-of-the-art CNNs or even Vision Transformers (if computationally feasible for the given dataset size) could provide a broader context for the achieved performance.

  • The "Why" Behind Kernel Modifications: While the paper mentions ConvNeXt inspiration for kernel modifications, a deeper discussion on why these specific changes (e.g., GeLU, fewer layers, LN) are beneficial in the context of rock classification (beyond general Transformer principles) could provide more actionable insights. For example, why GeLU might be better than ReLU for rock textures.

  • Training Details: Key training details such as the specific optimizer, learning rate schedule, batch size, and computational resources used are not explicitly mentioned beyond "training 10 epochs using learning rate := 1e-4". These details are crucial for reproducibility.

    Overall, this paper serves as a strong preliminary exploration for rock classification, offering practical solutions for data scarcity and architectural design in a specialized domain. The internal residual connection is a particularly noteworthy contribution that could find broader application in hybrid deep learning architectures.