The key idea behind low-rank factorization is to replace high-dimensional tensors with lower-dimensional tensors. One type of low-rank factorization is compact convolutional filters, where the over-parameterized (having too many parameters) convolution filters are replaced with compact
blocks to both reduce the number of parameters and increase speed.

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, SeparableConv2D

# Define input shape
input_shape = (64, 64, 3)  # For a 64x64 image with 3 color channels (RGB)

# Model with standard 3x3 convolutional filter
model_standard = Sequential([
    Conv2D(32, (3, 3), input_shape=input_shape, padding='same', activation='relu')
])

# Model with compact convolution using separable filters
model_compact = Sequential([
    SeparableConv2D(32, (3, 3), input_shape=input_shape, padding='same', activation='relu')
])

# Summary of both models
print("Standard Convolution Model Summary:")
model_standard.summary()

print("\nCompact Convolution (Separable) Model Summary:")
model_compact.summary()

GO ONE LEVEL DEEPER

The compression trade-off

Low-rank factorization replaces one large operation with smaller factors. The selected rank controls the balance between efficiency and approximation quality.

From a dense operation to compact factors
W

Original weights

A large matrix or convolution contains the full parameter set.

r

Factorize

Approximate the weights with two or more factors using rank r.

✓

Validate

Measure accuracy, latency, memory, and hardware behavior after conversion.

Keep in mind

  • Lower rank usually saves more compute but can lose more information.
  • Parameter reduction does not always translate directly to wall-clock speed.
  • Fine-tuning after factorization can recover part of the lost accuracy.

CONTINUE EXPLORING