Low Rank Approximation Implementation 3/4

Tech

Singular Value Decomposition (SVD) - Dimensionality Reduction

Author

Leila Mozaffari

Published

October 24, 2024

Image Compression

Singular Value Decomposition (SVD) - Dimensionality Reduction Single Image (PyTorch)

# !pip install torch torchvision

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
import numpy as np
from PIL import Image
import os

Download and Load the Dataset

from torchvision.datasets.utils import download_and_extract_archive

# Download Imagenette2-320
url = "https://s3.amazonaws.com/fast-ai-imageclas/imagenette2-320.tgz"
root = "./data"
download_and_extract_archive(url, download_root=root)

# Set dataset path
dataset_path = os.path.join(root, "imagenette2-320")

Downloading https://s3.amazonaws.com/fast-ai-imageclas/imagenette2-320.tgz to ./data\imagenette2-320.tgz

Extracting ./data\imagenette2-320.tgz to ./data

Define Data Transformations

We need to prepare our images for the model. We do this by resizing them to a uniform size and converting them into tensors, which is a format that PyTorch can understand.

# Define transformations with data augmentation
transform = transforms.Compose([
    transforms.Resize((224, 224)),          # Resize images to 224x224
    transforms.RandomHorizontalFlip(),      # Augment data with random horizontal flips
    transforms.RandomRotation(10),          # Augment data with random rotation
    transforms.ToTensor(),                  # Convert images to PyTorch tensors
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),  # Normalization
])

Load Dataset with DataLoader


# Load datasets
train_dataset = datasets.ImageFolder(root=os.path.join(dataset_path, 'train'), transform=transform)
valid_dataset = datasets.ImageFolder(root=os.path.join(dataset_path, 'val'), transform=transform)

# Create DataLoaders
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
valid_loader = DataLoader(valid_dataset, batch_size=32, shuffle=False)

Perform SVD for Dimensionality Reduction

To apply SVD, we will take a batch of images from the DataLoader, flatten them, and perform SVD on the flattened matrix.

Think of each image as a large grid of numbers. We want to compress these images while keeping the most important information. SVD breaks the image into three components that can be used to find a reduced version that still contains the important features.

Flatten Images: We reshape each image into a long row of numbers.
SVD: We apply SVD to the flattened images to find patterns in the data.
Retain Top k Features: We keep the top 100 components, which means we are compressing the data but keeping the main information.

# Get a batch of images and labels
images, labels = next(iter(train_loader))

# Flatten images to apply SVD
n_samples, c, h, w = images.shape
images_flat = images.view(n_samples, -1)  # Shape: (n_samples, c*h*w)

# Apply SVD
U, S, V = torch.svd(images_flat)

# Retain top k singular values for dimensionality reduction
k = 100  # Choose top k components
U_reduced = U[:, :k]  # Shape: (n_samples, k)

print(f'Original shape: {images_flat.shape}')
print(f'Reduced shape after SVD: {U_reduced.shape}')

Original shape: torch.Size([32, 150528])
Reduced shape after SVD: torch.Size([32, 32])

Train a Simple Neural Network on SVD-Reduced Features

Now that we have reduced features using SVD, let’s train a simple feed-forward neural network on these features.

Define Custom Dataset

class SVDImageDataset(torch.utils.data.Dataset):
    def __init__(self, loader, k):
        self.loader = loader
        self.data = []
        self.labels = []
        self.k = k  # Number of singular values/components to retain

        # Process each batch
        for images, labels in loader:
            for i in range(images.size(0)):
                # Flatten the image to 1D
                image_flat = images[i].view(-1).float()

                # Make the flattened image a 2D matrix for SVD application
                image_flat_matrix = image_flat.unsqueeze(0)  # Shape: (1, flattened_length)

                # Apply SVD on the image, consider it as a 1-row matrix
                U, S, V = torch.svd(image_flat_matrix)

                # Keep only the top `k` components (the first `k` values of U)
                if U.shape[1] >= k:
                    U_reduced = U[:, :self.k]  # Shape will be (1, k)
                else:
                    # If `U` has fewer components than `k`, pad it to ensure consistency
                    U_reduced = torch.cat([U, torch.zeros((U.shape[0], k - U.shape[1]))], dim=1)

                # Store the reduced representation
                self.data.append(U_reduced.view(-1))  # Flatten the matrix to have a size (k,)
                self.labels.append(labels[i])

        # Stack data and labels to create the dataset
        self.data = torch.stack(self.data)  # Shape: (number_of_images, k)
        self.labels = torch.tensor(self.labels)  # Shape: (number_of_images,)

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        return self.data[idx], self.labels[idx]

# Set the number of components to retain
k = 100

# Create reduced dataset using SVD
svd_train_dataset = SVDImageDataset(train_loader, k)
svd_valid_dataset = SVDImageDataset(valid_loader, k)

# Create DataLoaders
svd_train_loader = DataLoader(svd_train_dataset, batch_size=32, shuffle=True)
svd_valid_loader = DataLoader(svd_valid_dataset, batch_size=32, shuffle=False)

Define Neural Network

The model is a simple neural network with just a few layers. It takes the reduced data as input and learns to classify the images.

SimpleClassifier: A small neural network with a couple of layers.
Loss Function & Optimizer: We use cross-entropy loss and Adam optimizer to train the model.

# Define a simple fully connected neural network for classification
class SimpleClassifier(nn.Module):
    def __init__(self, input_size, num_classes):
        super(SimpleClassifier, self).__init__()
        self.fc1 = nn.Linear(input_size, 64)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(64, num_classes)

    def forward(self, x):
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)
        return x

# Initialize model, loss function, and optimizer
input_size = k  # We reduced the original dimensions to k with SVD
num_classes = len(train_dataset.classes)
model = SimpleClassifier(input_size, num_classes)

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

Train the Model

# Training loop
num_epochs = 100
for epoch in range(num_epochs):
    model.train()
    running_loss = 0.0
    for features, labels in svd_train_loader:
        # Move features and labels to the appropriate device
        features, labels = features.to(device), labels.to(device)

        # Zero the parameter gradients
        optimizer.zero_grad()

        # Forward pass
        outputs = model(features)
        loss = criterion(outputs, labels)

        # Backward pass and optimize
        loss.backward()
        optimizer.step()

        running_loss += loss.item()

    print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {running_loss/len(svd_train_loader):.4f}')

print("Training complete.")

Epoch [1/100], Loss: 2.3018
Epoch [2/100], Loss: 2.3018
Epoch [3/100], Loss: 2.3018
Epoch [4/100], Loss: 2.3018
Epoch [5/100], Loss: 2.3018
Epoch [6/100], Loss: 2.3018
Epoch [7/100], Loss: 2.3018
Epoch [8/100], Loss: 2.3018
Epoch [9/100], Loss: 2.3018
Epoch [10/100], Loss: 2.3018
Epoch [11/100], Loss: 2.3018
Epoch [12/100], Loss: 2.3018
Epoch [13/100], Loss: 2.3018
Epoch [14/100], Loss: 2.3018
Epoch [15/100], Loss: 2.3018
Epoch [16/100], Loss: 2.3018
Epoch [17/100], Loss: 2.3017
Epoch [18/100], Loss: 2.3018
Epoch [19/100], Loss: 2.3018
Epoch [20/100], Loss: 2.3017
Epoch [21/100], Loss: 2.3018
Epoch [22/100], Loss: 2.3018
Epoch [23/100], Loss: 2.3018
Epoch [24/100], Loss: 2.3018
Epoch [25/100], Loss: 2.3018
Epoch [26/100], Loss: 2.3018
Epoch [27/100], Loss: 2.3017
Epoch [28/100], Loss: 2.3018
Epoch [29/100], Loss: 2.3018
Epoch [30/100], Loss: 2.3017
Epoch [31/100], Loss: 2.3018
Epoch [32/100], Loss: 2.3018
Epoch [33/100], Loss: 2.3017
Epoch [34/100], Loss: 2.3018
Epoch [35/100], Loss: 2.3018
Epoch [36/100], Loss: 2.3018
Epoch [37/100], Loss: 2.3017
Epoch [38/100], Loss: 2.3018
Epoch [39/100], Loss: 2.3018
Epoch [40/100], Loss: 2.3018
Epoch [41/100], Loss: 2.3018
Epoch [42/100], Loss: 2.3018
Epoch [43/100], Loss: 2.3018
Epoch [44/100], Loss: 2.3018
Epoch [45/100], Loss: 2.3018
Epoch [46/100], Loss: 2.3018
Epoch [47/100], Loss: 2.3017
Epoch [48/100], Loss: 2.3018
Epoch [49/100], Loss: 2.3018
Epoch [50/100], Loss: 2.3017
Epoch [51/100], Loss: 2.3018
Epoch [52/100], Loss: 2.3018
Epoch [53/100], Loss: 2.3018
Epoch [54/100], Loss: 2.3017
Epoch [55/100], Loss: 2.3018
Epoch [56/100], Loss: 2.3018
Epoch [57/100], Loss: 2.3018
Epoch [58/100], Loss: 2.3018
Epoch [59/100], Loss: 2.3018
Epoch [60/100], Loss: 2.3018
Epoch [61/100], Loss: 2.3018
Epoch [62/100], Loss: 2.3017
Epoch [63/100], Loss: 2.3018
Epoch [64/100], Loss: 2.3018
Epoch [65/100], Loss: 2.3017
Epoch [66/100], Loss: 2.3018
Epoch [67/100], Loss: 2.3018
Epoch [68/100], Loss: 2.3018
Epoch [69/100], Loss: 2.3018
Epoch [70/100], Loss: 2.3018
Epoch [71/100], Loss: 2.3018
Epoch [72/100], Loss: 2.3018
Epoch [73/100], Loss: 2.3018
Epoch [74/100], Loss: 2.3018
Epoch [75/100], Loss: 2.3018
Epoch [76/100], Loss: 2.3017
Epoch [77/100], Loss: 2.3018
Epoch [78/100], Loss: 2.3017
Epoch [79/100], Loss: 2.3018
Epoch [80/100], Loss: 2.3017
Epoch [81/100], Loss: 2.3018
Epoch [82/100], Loss: 2.3018
Epoch [83/100], Loss: 2.3017
Epoch [84/100], Loss: 2.3018
Epoch [85/100], Loss: 2.3018
Epoch [86/100], Loss: 2.3017
Epoch [87/100], Loss: 2.3017
Epoch [88/100], Loss: 2.3018
Epoch [89/100], Loss: 2.3018
Epoch [90/100], Loss: 2.3018
Epoch [91/100], Loss: 2.3018
Epoch [92/100], Loss: 2.3018
Epoch [93/100], Loss: 2.3018
Epoch [94/100], Loss: 2.3017
Epoch [95/100], Loss: 2.3018
Epoch [96/100], Loss: 2.3018
Epoch [97/100], Loss: 2.3018
Epoch [98/100], Loss: 2.3017
Epoch [99/100], Loss: 2.3018
Epoch [100/100], Loss: 2.3018
Training complete.

Evaluate the Model

 #Validation loop
model.eval()
correct = 0
total = 0
with torch.no_grad():
    for features, labels in svd_valid_loader:
        outputs = model(features)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

print(f'Validation Accuracy: {100 * correct / total:.2f}%')

Validation Accuracy: 9.10%