Convolutional Neural Networks

Before motivating convolutions, it is worth understanding precisely why naive fully-connected (FC) networks fail on images.

Parameter explosion. A single FC layer mapping a 224×224 RGB image to 4096 hidden units needs 224×224×3×4096 ≈ 600 million parameters — just in the first layer. Training 600M parameters from scratch on ImageNet's 1.2M examples is a recipe for catastrophic overfitting. Even if we had the data, the memory and compute cost is prohibitive.

Ignoring spatial structure. An FC layer treats every pixel as interchangeable. It doesn't know that pixel (100,100) is spatially adjacent to (101,100). If you shuffle all the pixels of your training images with the same permutation, an FC network is completely unaffected. A cat's fur texture, an edge, a curve — all of these are local phenomena that span a small spatial neighborhood. FC layers cannot exploit this.

No translation invariance. If a cat in training images always sits in the center, an FC network learns weights specific to center-cat. A cat in the corner is a completely different input vector and the network must re-learn the concept from scratch. You'd need to show the network every object at every position — an impossibly large dataset.

The solution is to build the right inductive biases directly into the architecture. CNNs do exactly this with three ideas: local receptive fields, weight sharing, and spatial pooling.

import torch
import torch.nn as nn

# Fully-connected on 28x28 images: 784 -> 256 -> 10
fc_model = nn.Sequential(
    nn.Flatten(),            # 28x28 -> 784
    nn.Linear(784, 256),     # 784 * 256 = 200,704 params
    nn.ReLU(),
    nn.Linear(256, 10)       # 256 * 10 = 2,560 params
)
# Total: ~203K params, no spatial structure exploited

# CNN on same 28x28 images
cnn_model = nn.Sequential(
    nn.Conv2d(1, 32, kernel_size=3, padding=1),  # 3*3*1*32 = 288 params
    nn.ReLU(),
    nn.Conv2d(32, 64, kernel_size=3, padding=1), # 3*3*32*64 = 18,432 params
    nn.AdaptiveAvgPool2d(4),
    nn.Flatten(),
    nn.Linear(64*4*4, 10)                        # 10,240 params
)
# Total: ~29K params, fully exploits locality + translation

The word 'convolution' comes from signal processing, where it describes how a system's impulse response shapes an input signal. In CNNs we use a slightly different (actually cross-correlation) operation, but the name stuck.

1D intuition. Consider a 1D signal f[n] (e.g., audio samples) and a filter (kernel) g[k] of length K. The convolution output is:

(f * g)[n] = Σ_{k} f[n+k] · g[k]

The filter slides over the signal. At each position, we take a dot product between the filter and the local window. A filter g = [−1, 0, +1] computes a finite difference (edge in 1D); a Gaussian filter blurs; a box filter averages.

2D convolution for images. For a 2D image I[i,j] and kernel K[u,v] of size F×F:

(I * K)[i,j] = Σ_{u=0}^{F-1} Σ_{v=0}^{F-1} I[i+u, j+v] · K[u,v]

The kernel slides over every spatial position (i,j) of the image, computing a dot product each time. The result is a 2D feature map.

What kernels detect. A kernel is a pattern detector. The Sobel kernel [[-1,0,1],[-2,0,2],[-1,0,1]] detects vertical edges because its positive weights align with bright-on-left, dark-on-right patterns. A CNN learns its kernels from data — instead of hand-designing edge detectors, backpropagation discovers which patterns are predictive for the task.

Output spatial dimensions. Given input H×W, kernel F×F, padding P, stride S:

H_out = floor((H + 2P - F) / S) + 1
W_out = floor((W + 2P - F) / S) + 1

With F=3, P=1, S=1 ('same' padding): H_out = H, W_out = W — the output is the same size as the input.

Multi-channel convolution. Real images have C_in channels (3 for RGB). A single filter has shape F×F×C_in — it has a separate F×F slice for each input channel, and those slices are summed to produce one output channel. To produce C_out output channels, we use C_out such filters. Total parameters for one Conv2d layer: F × F × C_in × C_out + C_out (bias).

import numpy as np

def conv2d_naive(image, kernel, stride=1, pad=0):
    """2D cross-correlation (what PyTorch calls Conv2d)."""
    H, W = image.shape
    F = kernel.shape[0]  # assume square kernel
    if pad > 0:
        image = np.pad(image, pad, mode='constant')
    H_out = (H + 2*pad - F) // stride + 1
    W_out = (W + 2*pad - F) // stride + 1
    out = np.zeros((H_out, W_out))
    for i in range(H_out):
        for j in range(W_out):
            patch = image[i*stride:i*stride+F, j*stride:j*stride+F]
            out[i, j] = np.sum(patch * kernel)
    return out

# Vertical edge detector
image = np.array([
    [0, 0, 255, 255],
    [0, 0, 255, 255],
    [0, 0, 255, 255],
    [0, 0, 255, 255]
], dtype=float)

sobel_v = np.array([[-1, 0, 1],
                     [-2, 0, 2],
                     [-1, 0, 1]], dtype=float)

print(conv2d_naive(image, sobel_v, pad=1))
# Columns 1-2 will show large positive values: a vertical edge detected

The single most important property of convolutional layers is weight sharing: the same kernel is applied at every spatial position. This is both an enormous compression and a powerful inductive bias.

Parameter count comparison. For a 28×28 input with C_in=1 and we want C_out=32 feature maps:

• FC layer: 784 × (28×28×32) = 19M parameters
• Conv layer (3×3 kernel): 3×3×1×32 = 288 parameters

That 66,000× reduction comes entirely from weight sharing.

Translation equivariance. Formally, if T_d denotes shifting by d pixels, then:

Conv(T_d(x)) = T_d(Conv(x))

Shifting the input shifts the output by the same amount. This means: if the network learns to detect a vertical edge anywhere, it can detect vertical edges everywhere, using the same weights.

Note: equivariance ≠ invariance. Equivariance means the output shifts when the input shifts. Invariance (same output regardless of shift) comes later via pooling. CNNs achieve invariance hierarchically: equivariant feature maps + invariant pooling.

The weight sharing assumption. We are assuming the statistics of the data are spatially stationary — a vertical edge in the top-left has the same statistical structure as a vertical edge in the bottom-right. For natural images, this is overwhelmingly true. This is why weight sharing works: we are encoding a true fact about images into the architecture.

What would break it? Chest X-rays where the heart is always on the left might actually benefit from position-specific filters. Capsule Networks (Hinton, 2017) attempt to relax this assumption.

# Demonstrating equivariance numerically
import torch, torch.nn.functional as F

x = torch.zeros(1, 1, 6, 6)
x[0, 0, 2, 2] = 1.0  # single white pixel at center

kernel = torch.tensor([[[[0., -1., 0.],
                          [-1., 4., -1.],
                          [0., -1., 0.]]]])  # Laplacian

out1 = F.conv2d(x, kernel, padding=1)

# Now shift the input 1 pixel right
x_shifted = torch.roll(x, shifts=1, dims=3)
out2 = F.conv2d(x_shifted, kernel, padding=1)

# The output should also be shifted 1 pixel right
print(torch.allclose(torch.roll(out1, shifts=1, dims=3), out2))  # True

Convolutional layers give us equivariance — the feature maps move with the object. But for classification, we want the final answer to be the same whether the cat is at position (10,10) or (20,30). This is invariance. Pooling layers provide it.

Max pooling. The most common variant. Divide the feature map into non-overlapping F×F windows and take the maximum value in each. A 2×2 max pool halves both spatial dimensions.

Intuition: max pooling asks 'was this feature present somewhere in this region?' rather than 'was it present at this exact location?' It's a local argmax that discards precise position.

Average pooling. Takes the mean over the window. Less common in modern architectures for intermediate layers (max pooling is sharper/more discriminative) but used in global average pooling (GAP) at the end.

Global Average Pooling (GAP). Introduced by Lin et al. (2013) and popularized by ResNet. Takes the average over the entire spatial extent of each feature map, collapsing H×W×C to 1×1×C. This completely discards spatial information. Benefits:

1. No parameters (unlike a final FC layer)
2. Forces the network to put the class-relevant signal in the spatial average
3. Produces a compact representation resistant to spatial distortion

Stride as an alternative to pooling. Modern architectures (e.g., ResNet) often use strided convolutions (S=2) instead of explicit pooling layers. A strided convolution both computes features and downsamples, using the same learned weights. This is strictly more powerful (the network chooses how to downsample) but uses more parameters.

Spatial pyramid pooling (SPP). Pool at multiple scales simultaneously (e.g., 1×1, 2×2, 4×4 windows) and concatenate. Allows a fixed-size representation from variable-size inputs. Used in detection architectures (SPPNet, Faster R-CNN).

import torch
import torch.nn as nn

x = torch.tensor([[[[1., 3., 2., 4.],
                     [5., 6., 7., 8.],
                     [3., 2., 1., 0.],
                     [1., 2., 3., 4.]]]])  # 1x1x4x4

maxpool = nn.MaxPool2d(kernel_size=2, stride=2)
avgpool = nn.AvgPool2d(kernel_size=2, stride=2)
gap     = nn.AdaptiveAvgPool2d(1)  # Global Average Pool

print(maxpool(x))
# tensor([[[[6., 8.],
#           [3., 4.]]]])  -- took the max in each 2x2 block

print(avgpool(x))
# tensor([[[[3.75, 5.25],
#           [2.00, 2.00]]]])  -- took the mean

print(gap(x))
# tensor([[[[3.3125]]]])  -- mean of entire 4x4 = single number

After a convolutional layer, the output is called a feature map (or activation map). For C_out filters, we get C_out feature maps, each of size H_out×W_out. The collection is a 3D tensor (H_out × W_out × C_out).

Visualizing what features maps represent. Each channel of the feature map represents the response of one learned filter across all spatial positions. High activations mean 'this filter's pattern is present here.' Early layers learn Gabor-like filters (edges, textures). Middle layers combine these into parts (eyes, wheels). Late layers represent semantic concepts (faces, license plates).

Receptive field. The receptive field of a neuron is the region of the input image that can influence its value. This is a critical concept for understanding depth.

With a single 3×3 conv layer, each output neuron sees a 3×3 patch of the input. Add a second 3×3 conv layer: each output neuron depends on a 3×3 patch of the first layer's output, which itself covers a 3×3 patch — so the second-layer neuron has a 5×5 receptive field in the original image. After k layers of 3×3 convolutions: receptive field = (2k+1) × (2k+1).

Why stack many 3×3 instead of one large kernel? Two 3×3 conv layers have receptive field 5×5 but use 2×(3×3×C²) = 18C² parameters, while one 5×5 conv uses 25C² parameters. Three 3×3 layers give 7×7 receptive field with 27C² vs 49C². Smaller kernels are more parameter-efficient AND add more nonlinearities (one ReLU per layer), increasing expressive power. This is the key insight behind VGGNet.

Effective receptive field. Due to downsampling layers and the non-uniform way gradients flow, the effective receptive field (the region that actually influences activations strongly) is often much smaller than the theoretical one and has a Gaussian-like distribution. This is why very deep networks sometimes still struggle with global reasoning — deep layers may not effectively see the full image context. Attention mechanisms (Transformers) were partly motivated by this limitation.

def receptive_field_size(num_conv_layers, kernel_size=3, stride=1):
    """Compute theoretical receptive field after num_conv_layers
    of same kernel_size and stride (no pooling)."""
    rf = 1
    for _ in range(num_conv_layers):
        rf = rf + (kernel_size - 1) * stride
    return rf

for n in range(1, 10):
    print(f"{n} layers of 3x3: RF = {receptive_field_size(n)}x{receptive_field_size(n)}")

# Output:
# 1 layers of 3x3: RF = 3x3
# 2 layers of 3x3: RF = 5x5
# 3 layers of 3x3: RF = 7x7
# 5 layers of 3x3: RF = 11x11
# With 2x2 max pool (stride=2) inserted after layer 2:
# the effective RF doubles — this is how deep CNNs see large regions

Understanding architecture evolution reveals why each design choice exists.

LeNet-5 (LeCun, 1998). The original CNN, designed for handwritten digit recognition (MNIST). Architecture: Conv(5×5, 6 maps) → AvgPool(2×2) → Conv(5×5, 16 maps) → AvgPool → FC(120) → FC(84) → FC(10). Only ~60K parameters. Used sigmoid/tanh activations. Proved that end-to-end learned features beat hand-crafted ones for digits. The blueprint still holds.

AlexNet (Krizhevsky, 2012). Won ImageNet LSVRC 2012 by a historic 10.8% margin. Key innovations: (1) ReLU activations (instead of tanh) — avoids saturation, trains 6× faster; (2) Dropout (p=0.5) in FC layers — revolutionary regularization; (3) Data augmentation (random crops, flips, color jitter); (4) GPU training (2× GTX 580); (5) Local Response Normalization (now replaced by BatchNorm). Architecture: 5 conv layers + 3 FC layers, ~60M parameters.

VGGNet (Simonyan & Zisserman, 2014). Extreme depth with uniform architecture: only 3×3 conv filters (kernel_size=3, padding=1, stride=1) throughout, with 2×2 max-pool for downsampling. VGG-16 has 16 weight layers, VGG-19 has 19. Very simple, highly effective. Key lesson: depth > filter size, and uniform design is easier to reason about. Downside: 138M parameters, mostly in final FC layers — very memory-hungry.

GoogLeNet / Inception-v1 (Szegedy, 2014). Motivation: why choose kernel size? Use 1×1, 3×3, and 5×5 in parallel and concatenate. The Inception module applies multiple filter sizes simultaneously and concatenates their feature maps, letting the network choose. Also introduced 1×1 convolutions as bottlenecks — apply 1×1 conv to reduce channel count before expensive 3×3/5×5 convolutions. Result: 22 layers, only 5M parameters. Replaced final FC layers with GAP.

ResNet (He et al., 2015). The most important architecture innovation since LeNet. Introduces residual connections (skip connections): the input to a block is added to the block's output: y = F(x, {W_i}) + x. This reformulates the learning problem: instead of learning the desired mapping H(x) directly, the layers learn the residual F(x) = H(x) − x. If the optimal transformation is close to identity (often the case in deep networks), learning F(x) ≈ 0 is much easier than learning H(x) ≈ x through a stack of nonlinear layers. ResNet-50 has 50 layers and 25M parameters; ResNet-152 achieves 3.57% top-5 error on ImageNet. Residual connections also solve the vanishing gradient problem by providing a direct gradient highway through the identity shortcut.

import torch.nn as nn

# Residual Block (the heart of ResNet)
class ResidualBlock(nn.Module):
    def __init__(self, channels):
        super().__init__()
        self.conv1 = nn.Conv2d(channels, channels, 3, padding=1, bias=False)
        self.bn1   = nn.BatchNorm2d(channels)
        self.relu  = nn.ReLU(inplace=True)
        self.conv2 = nn.Conv2d(channels, channels, 3, padding=1, bias=False)
        self.bn2   = nn.BatchNorm2d(channels)

    def forward(self, x):
        identity = x                          # Skip connection
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out = out + identity                  # Add residual
        return self.relu(out)

# Key insight: gradient flows directly through the '+' operation
# d(out)/d(x) = d(F(x))/d(x) + 1  -- the '+1' prevents vanishing gradients

Batch Normalization (Ioffe & Szegedy, 2015) is the most impactful training trick in deep learning history. It is applied between the convolutional layer and the activation function (or sometimes after), and it transformed what was possible to train.

The problem: internal covariate shift. As network weights update during training, the distribution of activations at each layer constantly shifts. Layer l+1 must continuously re-adapt to a changing input distribution. This slows training and requires careful initialization and small learning rates.

What BatchNorm does. For a mini-batch B = {x₁, ..., xₘ}, compute:

1. Mini-batch mean: μ_B = (1/m) Σ xᵢ
2. Mini-batch variance: σ²_B = (1/m) Σ (xᵢ − μ_B)²
3. Normalize: x̂ᵢ = (xᵢ − μ_B) / √(σ²_B + ε)
4. Scale and shift: yᵢ = γ · x̂ᵢ + β

γ and β are learned parameters — the network can learn to undo the normalization if needed, so BatchNorm adds no expressive power restriction.

Effects in practice:

• Allows much higher learning rates (10-100× larger)
• Makes training much less sensitive to initialization
• Acts as a regularizer (the noise from mini-batch statistics adds regularization, sometimes reducing the need for dropout)
• Accelerates convergence dramatically — often 10-14× fewer training iterations

At test time: instead of using mini-batch statistics (which may be a single image), BatchNorm uses running estimates of μ and σ² accumulated during training. This is why model.eval() in PyTorch matters — it switches BatchNorm to use running statistics.

For convolutional layers: normalization is per-channel, across the batch and spatial dimensions (H×W). A single γ and β per channel.

Limitations: BatchNorm is batch-size-dependent and performs poorly with very small batches (batch size < 8). Layer Normalization (normalizes per sample across channels) is preferred in Transformers and small-batch settings.

import torch
import torch.nn as nn

# Standard CNN block with BatchNorm
class ConvBNReLU(nn.Module):
    def __init__(self, in_ch, out_ch, kernel=3, stride=1, padding=1):
        super().__init__()
        self.block = nn.Sequential(
            nn.Conv2d(in_ch, out_ch, kernel, stride, padding, bias=False),
            # bias=False because BatchNorm's beta already shifts the mean
            nn.BatchNorm2d(out_ch),  # learnable gamma, beta per channel
            nn.ReLU(inplace=True)
        )

    def forward(self, x):
        return self.block(x)

# Demonstrate train vs eval mode effect on BatchNorm
bn = nn.BatchNorm2d(4)
bn.train()   # uses mini-batch mean/var + accumulates running stats
bn.eval()    # uses running_mean / running_var (accumulated during training)

The 1×1 convolution seems paradoxical — a kernel that looks at only one pixel at a time. But it is one of the most powerful ideas in CNN design (introduced formally by Lin et al., 'Network in Network', 2013).

What a 1×1 conv does. A 1×1 convolution with C_in input channels and C_out output channels applies a learned linear combination across channels at each spatial location independently. It is equivalent to a fully-connected layer applied pointwise across the spatial grid.

Use case 1: channel reduction. A 256-channel feature map → 64-channel via 1×1 conv. This is a bottleneck: it compresses the channel dimension before an expensive 3×3 conv, reducing compute by 4×. The 3×3 conv then operates on 64 channels instead of 256.

Use case 2: adding nonlinearity without changing spatial size. Stack multiple 1×1 convolutions with ReLU to increase depth (and expressivity) without changing the spatial dimensions or receptive field.

Use case 3: matching dimensions for residual connections. When the input and output of a residual block have different channel counts (e.g., 64 → 128), a 1×1 conv with stride 2 is used as the shortcut connection to match shapes.

The Bottleneck Block (ResNet-50+). For deep ResNets, the full residual block uses: 1×1 conv (reduce channels) → 3×3 conv → 1×1 conv (expand channels). E.g., 256 → 64 → 64 → 256. The 3×3 conv is cheap because it operates on the compressed 64-channel representation. This allows building much deeper networks at the same compute budget.

class BottleneckBlock(nn.Module):
    """ResNet-50 style bottleneck: 1x1 -> 3x3 -> 1x1"""
    expansion = 4

    def __init__(self, in_ch, mid_ch):
        super().__init__()
        out_ch = mid_ch * self.expansion  # e.g., 64 * 4 = 256

        self.layers = nn.Sequential(
            # 1x1: reduce channels (no spatial mixing)
            nn.Conv2d(in_ch, mid_ch, 1, bias=False),
            nn.BatchNorm2d(mid_ch),
            nn.ReLU(inplace=True),
            # 3x3: spatial feature extraction on compressed rep
            nn.Conv2d(mid_ch, mid_ch, 3, padding=1, bias=False),
            nn.BatchNorm2d(mid_ch),
            nn.ReLU(inplace=True),
            # 1x1: expand channels back
            nn.Conv2d(mid_ch, out_ch, 1, bias=False),
            nn.BatchNorm2d(out_ch),
        )
        # Shortcut must match output channels
        self.shortcut = nn.Sequential(
            nn.Conv2d(in_ch, out_ch, 1, bias=False),
            nn.BatchNorm2d(out_ch)
        ) if in_ch != out_ch else nn.Identity()

        self.relu = nn.ReLU(inplace=True)

    def forward(self, x):
        return self.relu(self.layers(x) + self.shortcut(x))

Understanding why CNNs work is as important as knowing how they are structured. The answer lies in the combination of the right inductive biases and the compositional nature of visual patterns.

Zeiler & Fergus (2013) — visualizing what CNNs learn. Using deconvolution networks, they visualized which input patterns maximally activate each filter in AlexNet. The findings were striking and have since been confirmed many times:

• Layer 1–2: Gabor-like filters (oriented edges, color blobs). The network independently rediscovers the same features that neuroscientists find in V1 of the visual cortex and that computer vision researchers spent decades hand-crafting.
• Layer 3: Textures, grids, simple curves and junctions.
• Layer 4: Parts — dog faces, text wheels, bird legs. Features start becoming class-specific.
• Layer 5 and FC: Complete objects, faces, specific breeds.

The compositionality argument. Natural images are hierarchically compositional: pixels form edges, edges form textures and shapes, shapes form parts, parts form objects. CNNs exploit this by building their own hierarchy — each layer learns to detect patterns made of lower-level patterns. This matches the true structure of the data.

Transfer learning as evidence. If CNN features were arbitrary, they wouldn't transfer between tasks. But features learned on ImageNet classification transfer remarkably well to medical imaging, satellite analysis, and fine-grained classification — even when the domains seem visually unrelated. This strongly suggests the early layers have learned universal visual representations.

The role of nonlinearity. ReLU (and other activations) between conv layers is essential. Without nonlinearities, stacked linear layers collapse to a single linear transformation — no matter how many layers you add. Nonlinearities allow the network to represent complex, non-linear decision boundaries in input space.

Max pooling builds 'part-based' representations. By taking the max over a spatial region, the network learns 'is this part present anywhere nearby?' — a form of local spatial invariance that allows slight misalignments of parts without breaking the detector. This is the key to robust recognition.

# Visualize filters learned by a trained model
import torchvision.models as models
import matplotlib.pyplot as plt
import numpy as np

resnet = models.resnet18(weights='DEFAULT')

# First conv layer: 64 filters of shape 7x7x3
filters = resnet.conv1.weight.data  # shape: [64, 3, 7, 7]

# Normalize each filter for visualization
fig, axes = plt.subplots(8, 8, figsize=(10, 10))
for i, ax in enumerate(axes.flat):
    f = filters[i].permute(1, 2, 0).numpy()  # CxHxW -> HxWxC
    f = (f - f.min()) / (f.max() - f.min())  # normalize to [0,1]
    ax.imshow(f)
    ax.axis('off')
plt.suptitle('ResNet-18 Layer 1 Filters (7x7x3, 64 total)')
# You will see oriented edge detectors and color-opponent filters
# that closely resemble simple cells in the primary visual cortex (V1)

Standard convolutions are computationally expensive. For mobile and embedded applications, we need networks that are fast and small. Depthwise separable convolutions, introduced in MobileNet (Howard et al., 2017), achieve near-standard accuracy at a fraction of the cost.

Standard convolution cost. For a layer with C_in input channels, C_out output channels, F×F kernel, and output of size H×W:

• Multiplications: H × W × C_in × C_out × F × F

Depthwise separable convolution. Factor the standard convolution into two steps:

1. Depthwise conv: Apply a separate F×F filter to each input channel independently — no cross-channel mixing. Produces C_in output channels. Cost: H × W × C_in × F × F
2. Pointwise conv (1×1 conv): Apply a 1×1 conv with C_out filters to mix channels. Cost: H × W × C_in × C_out

Total cost ratio:

(HW·C_in·F² + HW·C_in·C_out) / (HW·C_in·C_out·F²)
= 1/C_out + 1/F²
≈ 1/F² (when C_out >> 1)

For 3×3 filters: ~8-9× cheaper. For 5×5: ~25× cheaper.

Where is the cost? The standard conv mixes spatial and channel information simultaneously. Depthwise separable factorization separates these two operations — spatial filtering (depthwise) and channel mixing (pointwise). The insight is that these can be done independently without much loss in expressivity.

MobileNetV2 adds inverted residuals with linear bottlenecks: expand channels (1×1), apply depthwise conv (3×3), project back down (1×1), add skip connection. This is the workhorse of mobile vision today.

class DepthwiseSeparableConv(nn.Module):
    """~8-9x cheaper than standard Conv2d (3x3 case)."""
    def __init__(self, in_ch, out_ch, stride=1):
        super().__init__()
        self.depthwise = nn.Conv2d(
            in_ch, in_ch,
            kernel_size=3, stride=stride, padding=1,
            groups=in_ch,   # KEY: one filter per input channel
            bias=False
        )
        self.pointwise = nn.Conv2d(
            in_ch, out_ch,
            kernel_size=1,  # 1x1 to mix channels
            bias=False
        )
        self.bn1 = nn.BatchNorm2d(in_ch)
        self.bn2 = nn.BatchNorm2d(out_ch)
        self.relu = nn.ReLU6(inplace=True)  # MobileNet uses ReLU6

    def forward(self, x):
        x = self.relu(self.bn1(self.depthwise(x)))
        x = self.relu(self.bn2(self.pointwise(x)))
        return x

# Compare parameters:
standard = nn.Conv2d(32, 64, 3, padding=1)  # 32*64*9 = 18,432 params
ds       = DepthwiseSeparableConv(32, 64)   # 32*9 + 32*64 = 2,336 params
print(f"Standard: {sum(p.numel() for p in standard.parameters())} params")
print(f"DS:       {sum(p.numel() for p in ds.parameters())} params")