2024. Edge-based Generative AI for Image Processing Aplication Implementing over-the-air (OTA) updates for edge devices that run large language models (LLMs) as part of their operations (referred to here as “LLM ops”)

On-device processing: Prioritizing privacy and accessibility, LLMs on-device processing, ensuring that your data remains on your device. This not only enhances security but also has the potential to minimize latency, contributing to a more efficient user experience.

Generative AI models are a class of AI that can generate new data that resembles existing data. They include foundation models that can be fine-tuned for different tasks, Variational Autoencoders (VAEs) that reduce the dimensionality of data and Generative Adversarial Networks (GANs) that use competing networks to generate realistic samples. Other models include Transformer-based models that use attention mechanisms to model long-term text dependencies, diffusion models that address information decay by removing noise in the latent space, and multimodal large language models (LLMs) that integrate image and text understanding. In summary, generative AI models are a class of AI that can generate new data that resembles existing data. They include foundation models, VAEs, GANs, Transformer-based models, diffusion models, and multimodal LLMs.

Farshid PirahanSiah

OTA Updates for Edge Devices with LLM Ops

  • Deployment Architecture
    • Containerized
      • Use Docker for easy update rollout.
      • Restart apps with new container images.
    • Microservices
      • Independently update smaller services.
  • Version Control and Rollback
    • Version Tracking
      • Assign unique IDs to each update.
    • Rollback Mechanism
      • Revert to previous version if update fails.
  • Security
    • Secure Channels
      • Use HTTPS or MQTT over TLS.
    • Authentication and Integrity
      • Employ digital signatures for updates.
  • Update Process
    • Minimal Downtime
      • Delta Updates
        • Send only differences to reduce data needs.
      • Staged Rollouts
        • Update devices in stages to minimize risks.
    • Automation
      • Scheduled Updates
        • Plan updates during low-usage times.
      • Monitoring
        • Automatically monitor and log the update process.
  • User Interaction
    • Notifications
      • Inform about upcoming updates.
    • Feedback Mechanism
      • Provide channels for user feedback on updates.
  • Testing and Validation
    • Simulated Environment Testing
      • Test updates in a controlled setting.
    • Field Testing
      • Conduct testing on a small set of devices before full rollout.

Generative AI models on Edge with on-device training

  • CNN, RNN, Transformer-based models, LLMs (GPT, )
  • Generative AI models
    • foundation model
      • train on huge data -> adapt to applications
      • GPT, CLIP, DALL-E,
    • Variational Autoencoders (VAEs)
      • rapidly reduce the dimensionality of samples
      • input (image) -> encoder -> latent space -> decoder -> reconstructed output (image)
      • use for
        • image synthesis
        • data compression
        • anomaly detection
      • Standard VAEs
        • Beta-VAE: Balances latent capacity and reconstruction.
      • Conditional VAEs (CVAEs)
        • Conditional Beta-VAE: Adds conditional variables for targeted generation.
      • Hierarchical VAEs
        • NVAE: Deep hierarchy, residual connections for stable high-quality output.
    • Generative Adversarial Networks (GANs)
      • use competing networks to produce realistic samples
      • generator ; discriminator
      • GANs in finance, spaceGAN (geospatial data), styleGAN2 (create video game characters)
      • Convolutional GANs
        • DCGAN: Deep convolutional networks for stable, quality generation.
      • Style-based GANs
        • StyleGAN: Fine-grained style control, realistic faces.
        • StyleGAN2: Redesigns normalization, reduces artifacts.
      • Progressive Growing GANs
        • PGGAN: Increases network size during training for high-resolution output.
    • Transformer-based Models
      • use attention mechanisms to model long-term text dependencies
      • Autoregressive Models
        • GPT-3: Predictive text generation based on input.
        • Image GPT (iGPT): Generates images pixel by pixel.
      • Multimodal Transformers
        • DALL-E: Generates images from text descriptions.
    • Diffusion Models
      • address information decay by removing noise in the latent space
      • step 1: forward diffusion to add random noise to the data;
      • step 2: reverse diffusion: turn the noise , recover data, generate the desired output
      • Basic Diffusion Models
        • DDPM: Generates data from noise by reversing diffusion.
      • Advanced Diffusion Techniques
        • Improved DDPM: More efficient, improved sampling.
        • Guided Diffusion: Produces specific outputs with classifier guidance.
    • Multimodal Large Language Models (LLMs)
      • CLIP
        • Integrates image and text understanding.
      • Perceiver IO
        • Handles audio, visual, text with a generalized architecture.
      • FLAVA
        • Self-supervised, unified model for vision, language, and multimodal tasks.
  • LLMOps: MLOps Tools: MLflow and Hugging Face

    • 📚 🤔 MLflow →

      • MLflow Tracking - Logs key metrics, parameters, models, and other artifacts when running ML code to monitor experiments
      • MLflow Projects - Configurable standard format for organizing ML code to ensure consistency and reproducibility
      • MLflow Models - Package ML model files with their dependencies so they can be deployed on diverse platforms
      • pip install mlflow
      • mlflow ui
      • mlflow experiments create –experiment-name metrics-test ==> id 2
      • MLFLOW_EXPERIMENT_ID=2 python test-mlflow.py
      • load_model= mlflow.pyfunc.load_model(logged_model)
      • mlflow run . -P filename=inputfile
      • hugging face
        • models, dataset, spaces
        • streamlit, gradio
        • pip install huggingface_hub
        • huggingface-cli login
        • git lfs install
        • git clone
        • GGUF-Quantization-of-any-LLM
          • MiniCPM-V and OmniLMM are open-source multimodal large models designed for image and text understanding
          • ggml
          • git clone https://github.com/ggerganov/llama.cpp
          • cd llama.cpp && LLAMA_CUBLAS=1 make && pip install -r requirements/requirements-convert-hf-to-gguf.txt
          • for m in methods: qtype = f”{quantized_path}/{m.upper()}.gguf” os.system(“./llama.cpp/quantize “+quantized_path+”/FP16.gguf “+qtype+” “+m)
          • ./llama.cpp/main -m ./quantized_model/Q4_K_M.gguf -n 90 –repeat_penalty 1.0 –color -i -r “User:” -f llama.cpp/prompts/chat-with-bob.txt
          • type 0 quant > w=d*q
          • type 1 quant > w=d*q+m
          • 4-bit
          • Q2_K
        • datasets
        • fastAPI, uvicorn,
        • docker build -t huggingface:local .
        • docker run -i -p 8000:8000 huggingface:local
        • in url /docs -> put your text input
        • CI/CD github action
        • code
        • fine-tune
          • transfer learning
          • ONNX on hugging face

          • Llama cpp

  • Size
    • 7B = 7 B * 4 bytes = 28 GB
    • 14B = 14 B * 4 bytes = 56 GB
    • 70B = 70 B * 4 bytes = 280 GB
  • model
    • tokenizer
      • The role of tokenization then is to convert the input text into the numeric values and vice versa
      • each model has own tokenization format
      • llama2 is 32000 token large

  • Memory Reduction Techniques

    Technique Description RAM Reduction Notes
    Quantization Reduces the number of bits used to represent weights and activations. Significant Popular options include 8-bit (int8), 4-bit, and 1-bit. Accuracy might degrade slightly with lower bit widths.
    Pruning Removes redundant or unimportant weights from the model. Moderate Can lead to sparsity in the weight matrix, allowing for compressed storage.
    Knowledge Distillation Trains a smaller student model to mimic the behavior of a larger teacher model (LLAMA2 70B). Significant The student model will have lower memory requirements, but might not achieve the same performance as the teacher.
    Model Parallelism Distributes the model across multiple GPUs, each holding a portion of the weights. N/A (reduces memory per device) Requires specialized hardware and software for efficient communication between GPUs.
    Adaptive Quantization (2024) Tailors the bit width for each weight based on its importance, achieving a better balance between memory reduction and accuracy. Potentially higher Expected to be a key advancement in 2024 for memory-efficient model deployment.
    Hierarchical Quantization (2024) Applies different quantization levels to different parts of the model based on their sensitivity to precision loss. Potentially significant A promising technique in research for further memory savings while preserving accuracy.
    Transformer Sparsification (2024) Explores techniques like weight clustering and structured pruning specifically for Transformer architectures like LLAMA2. Moderate to high An active research area with potential for memory reduction in large language models.

  • RAM Requirements with Reduction Techniques (Estimates)

    Technique RAM Reduction Estimated RAM Notes
    Quantization (8-bit) 2x 70 GB Achievable with minimal accuracy loss.
    Quantization (4-bit) 4x 35 GB May require specific hardware support and potentially more accuracy loss.
    Quantization (1-bit) 8x 17.5 GB Significant accuracy drop expected. Use with caution.
    Pruning (moderate) 20-30% 112-98 GB Effectiveness depends on pruning strategy and impact on accuracy.
    Knowledge Distillation (smaller student model) 70-90% 14-42 GB Student model performance depends on teacher model and training strategy.
    Adaptive Quantization (2024) Up to 3x 47-56 GB Expected to offer better accuracy-efficiency trade-off.
    Hierarchical Quantization (2024) 3x-5x 28-47 GB Research is ongoing, effectiveness depends on implementation.
    Transformer Sparsification (2024) 20-40% 84-112 GB An active research area, estimates may vary.
  • other
    • quantize 32F to 16, 8,4,2,1
    • float to int

Large Vision Models (LVMs)

  • Stable Diffusion
    • Why Stable Diffusion
      • Brainstorming and Ideation: Rapidly generate images to explore various design layouts, scenes, or concepts. It’s an efficient tool for experimenting with creative ideas, such as different settings or stylized visual effects, providing more specific and customized placeholders than stock photos or traditional sketches.
      • Character Design: Whether for video games, animations, or comic books, Stable Diffusion can help conceptualize and visualize characters, aiding in the development and refinement of unique designs.
      • Storyboarding: Create visual narratives for video productions, movies, or advertisements. Stable Diffusion can generate scenes and sequences, facilitating the storytelling process and the visualization of cinematic ideas.
      • Communication: Enhance presentations or proposals with high-quality visualizations. For office managers, interior designers, or event planners, it’s a tool to convey ideas about space layouts, mood, atmosphere, and color schemes effectively, even if the images aren’t perfect replicas.
      • Asset Generation: Produce elements for compositing into photographs or artworks, create textures for 3D modeling, or develop backgrounds for motion graphics. It’s a versatile tool for augmenting traditional and digital art processes with unique visual components.
      • Fun and Exploration: Beyond professional applications, Stable Diffusion offers a platform for fun and creative exploration. It encourages playful experimentation with images and concepts, often leading to unexpected and delightful results.
    • Stable Diffusion
      • Image Style
        • Blends photorealism and illustration
        • Customizable styles via data models
      • Data Models
        • Open-source, community-driven enhancements
        • Customizable for content safety or style
      • Accessibility
        • Free and open-source
        • Standalone applications
        • Integrates with software like Photoshop
      • Ethics
        • Training on diverse datasets
        • Ethical considerations similar to competitors
      • Features
        • Image-to-image generation
        • Custom training for specific recognition
      • Compared to Others
        • DALL-E and Midjourney focus on photorealism
        • Adobe Firefly trained on ethical datasets
        • Midjourney accessible via Discord
        • Others require payment, Stable Diffusion is free
      • Community Impact
        • Vibrant open-source community
        • Continuous improvement and feature additions
      • Technology
        • AI image generator revolution
        • Different from traditional digital tools
        • Democratizes sophisticated image generation
  • Generative Models
    • History
      • Evolution of Image Generation Models:
        • The development of image generation technologies has evolved from GANs and VAEs to include flow-based models, each contributing to the field’s growth.
        • Despite their successes, earlier models like GANs often faced challenges related to training stability and the diversity of generated outputs.
      • Introduction to Diffusion Models:
        • Diffusion models represent a breakthrough approach, inspired by the principles of non-equilibrium thermodynamics.
        • They function by gradually introducing noise to an image (forward process) and then learning to reverse this process (reverse diffusion) to generate images from noise, achieving high-quality and diverse outputs.
      • Mechanics of Unconditional Image Generation:
        • Noise Vector: The generation process starts with a noise vector, which serves as a randomized seed for image creation.
        • Training: Through exposure to a diverse dataset, diffusion models learn the underlying distribution of the training images.
        • Generation: By reversing the diffusion process, these models can create new images that are independent of any specific external input, relying solely on the learned patterns.
      • Applications and Impact:
        • Artistic Creativity: Artists and designers leverage diffusion models to explore new aesthetic realms and generate unique artworks.
        • AI Training Data Augmentation: Diffusion models enhance the robustness of AI systems by generating varied and realistic training datasets.
        • Virtual Reality: They contribute to creating more immersive and realistic environments within VR applications.
        • Medical Imaging: Advances in generating detailed and accurate medical images for research and diagnostic purposes.
    • Generative
      • Generative vs. Discriminative Models:
        • Discriminative Models: Aim to categorize or predict outcomes by learning the decision boundary between different classes within the dataset. Examples include regression and classification models.
        • Generative Models: Focus on learning the underlying distribution of a dataset to generate new, plausible data samples that resemble the original dataset. They are probabilistic and can produce diverse outputs for the same input.
      • Understanding Generative Models:
        • Generative models are trained on a dataset to understand its distribution. After training, these models can generate new data points that, while not identical to, are similar to those in the training set.
        • They are primarily used in unsupervised learning settings, where the goal is to model the structure or distribution of data rather than predict labels.
      • Key Features of Generative Models:
        • Data Generation: Capable of producing new data instances that mimic the learned data distribution.
        • Unsupervised Learning: Often do not require labeled data, enabling them to learn from datasets without explicit annotations.
        • Probabilistic Nature: Introduce randomness in the generation process, allowing for the creation of varied and unique outputs.
      • Types of Generative Models:
        • Generative Adversarial Networks (GANs): Comprise two neural networks, a generator and a discriminator, that are trained simultaneously in a competitive setting where the generator aims to produce data indistinguishable from real data, and the discriminator strives to differentiate between real and generated data.
        • Diffusion Models: A newer class of generative models that gradually learn to reverse a diffusion process (a process that adds noise to data) to generate data from noise, effectively learning the data distribution in reverse.
      • Applications:
        • Image Generation: Creating realistic images, artwork, or photorealistic edits.
        • Natural Language Processing: Generating coherent text, dialogue, or completing sentences in an autocompletion system.
        • Data Augmentation: Generating additional training data for machine learning models, especially in scenarios where collecting real data is impractical or expensive.
        • Anomaly Detection: Modeling normal behavior to identify deviations or anomalies within new data.
      • Probabilistic vs. Deterministic Models:
        • Generative models are inherently probabilistic, incorporating randomness to generate diverse outputs, as opposed to deterministic discriminative models that produce consistent outputs for given inputs.
    • Good for:
      • Upsampling Imbalanced Datasets:
        • Problem: Many real-world datasets suffer from class imbalance, where certain classes are underrepresented compared to others, leading to biased models.
        • Solution: Generative models can create synthetic samples of the underrepresented classes, balancing the dataset and enabling more equitable model training outcomes.
      • Imputation of Missing Values:
        • Problem: Datasets often have missing entries that can skew analysis and model training, reducing the overall quality and reliability of the predictions.
        • Solution: Generative models can intelligently fill in missing values by predicting plausible data points based on the learned distribution, preserving the dataset’s integrity.
      • Anonymizing Sensitive Datasets:
        • Problem: Using real, sensitive data for training can risk privacy breaches and non-compliance with data protection regulations.
        • Solution: Generative models can create anonymized datasets that maintain statistical properties of the original data but do not link back to any real individual, enhancing privacy protection.
      • Image and Video Enhancement:
        • Application: Improving the resolution and quality of images and videos, commonly referred to as super-resolution, which is beneficial for medical imaging, surveillance, and entertainment.
      • Synthetic Data Generation for Training:
        • Application: Generating realistic, varied datasets for training machine learning models where collecting real-world data is impractical, expensive, or impossible.
      • Drug Discovery and Molecular Modeling:
        • Application: Creating new molecular structures for potential drugs, accelerating the discovery process by exploring chemical space more efficiently than traditional methods.
      • Content Creation:
        • Application: Producing art, music, written content, and other creative works, opening new avenues for creativity and automating aspects of content production.
      • Voice Generation and Modification:
        • Application: Generating or altering voice recordings for applications like virtual assistants, dubbing in different languages, or restoring damaged audio recordings.
      • Financial Modeling:
        • Application: Simulating market conditions, customer behavior, or risk scenarios, aiding in strategic planning and risk management.
    • Methods
      • GAN
        • Generative Adversarial Networks (GANs):
        • Structure: Consists of two neural networks—the Generator and the Discriminator—operating in a zero-sum game framework where the success of one network signifies a setback for the other.
        • Generator: Aims to generate new data points (often images) that are indistinguishable from genuine data points in the training dataset. It learns to produce outputs that the Discriminator cannot easily classify as fake.
        • Discriminator: Acts as a classifier, trying to distinguish between real data points from the dataset and fake ones produced by the Generator. Over time, it becomes better at identifying generated data points.
        • Training Dynamics: Through their adversarial training process, both networks improve in their functions—the Generator produces increasingly realistic data, while the Discriminator becomes better at detecting nuances that distinguish real from fake.
        • Applications: GANs have been successfully applied in various domains, including image and video synthesis, style transfer, data augmentation, and more.
        • Foundational Paper: Introduced by Ian Goodfellow and colleagues in 2014, providing a novel framework for unsupervised machine learning.
      • Diffusion
        • Denoising Diffusion Probabilistic Models (DDPMs):
        • Approach: Utilizes a forward diffusion process to gradually add noise to data until it becomes indistinct from random noise, then learns a reverse diffusion process to reconstruct the original data from the noise.
        • Key Process: The model iteratively denoises the data, effectively learning the data distribution by reversing the noise addition process. This method requires a deep understanding of the data’s structure to effectively remove the noise and restore the original content.
        • Training Requirement: Like GANs, DDPMs are trained using unsupervised learning techniques, allowing them to generate high-quality images without needing labeled datasets.
        • Emergence and Applications: Though newer than GANs, diffusion models have rapidly gained attention for their ability to generate highly detailed and realistic images, finding applications in areas similar to those of GANs, including creative content generation and enhancement.
        • Foundational Paper: Introduced by Jonathan Ho et al. in 2020, marking a significant advancement in the field of generative modeling.
  • attention-based models
    • limitation of Feedforward
      • Feedforward Neural Networks Overview:
        • Structure: Composed of neurons arranged in layers, where each neuron in one layer connects to neurons in the subsequent layer, facilitating a one-directional flow of information from input to output.
        • Types: Includes dense (fully connected) and convolutional neural networks, commonly applied in various machine learning tasks.
        • Operation: Processes inputs to outputs directly, with each layer’s output serving as the next layer’s input, culminating in a prediction.
      • Characteristics:
        • Dense vs. Sparse Connections: Networks can be densely connected (every neuron to every neuron in the next layer) or sparsely connected (selective connections), impacting complexity and computational load.
        • Static Input Handling: Designed to process entire data points at once, with each input treated as an independent, isolated instance without interrelationships or sequence.
        • No Feedback Loops: Outputs are not recycled back into the network; each output is final for its given input.
      • Limitations in Handling Sequential Data:
        • Lack of Temporal Dynamics: Unable to inherently capture or utilize the time-based relationships present in sequential data due to the absence of mechanisms to remember previous inputs or states.
        • Order Insensitivity: The significance of data point order in sequences, crucial for understanding context and changes over time, is not natively accounted for.
        • Inefficiency with Sequences: While they can theoretically model sequences by increasing network width and depth, this approach is computationally inefficient and lacks the nuanced understanding achievable with models designed for sequence processing.
      • Sequential Data Challenges:
        • Language: The meaning derived from words and sentences depends significantly on their order, requiring a model that understands context and sequence to interpret or generate language correctly.
        • Time-Series Data: Financial data, sensor readings, and other time-series datasets contain valuable temporal patterns that feedforward networks cannot exploit effectively.
        • Video and Audio Processing: Sequences of images (video) or sound waves (audio) necessitate understanding the progression and changes over time, beyond the capability of simple feedforward architectures.
      • Alternative Approaches:
        • Recurrent Neural Networks (RNNs): Specifically designed to handle sequential data by maintaining a form of memory about previous inputs through internal states, addressing the temporal dynamics limitation.
        • Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU): Variants of RNNs that effectively capture long-range dependencies in data, overcoming some of RNNs’ limitations, such as the vanishing gradient problem.
    • RNN
      • RNN Structure and Operation:
        • Memory Through Hidden States: Unlike feedforward neurons, recurrent neurons possess a hidden state (h) that acts as memory, retaining information from previous inputs in the sequence.
        • Feedback Loops: The output at each time step (t) is fed back into the network as part of the input for the next time step (t+1), allowing the network to make decisions based on past information.
        • Weighted Inputs and States: Each input (X) and the previous hidden state (h[t-1]) are weighted by their respective weight matrices (Wx for inputs and Wy for hidden states) before being combined and processed to produce a new hidden state and output.
      • Unrolling Through Time:
        • Sequential Processing: To handle inputs at different time steps, RNNs “unroll” through time, processing each element of the sequence step-by-step, updating the hidden state with each iteration.
        • Length of Sequence: The extent to which an RNN is unrolled corresponds to the length of the input sequence, ensuring each input is accounted for in the processing.
      • RNN Layers:
        • RNN Cell: A single layer in an RNN is often referred to as an RNN cell, comprising multiple recurrent neurons. The architecture of an RNN involves unrolling these cells to form the network’s recurrent layers.
        • Layer Construction: The number of layers (or depth) of an RNN is determined by the number of RNN cells (or neurons) and how these cells are arranged to process the sequence.
      • Advantages and Applications:
        • Temporal Dynamics: RNNs excel in capturing time-dependent patterns, making them suitable for tasks where the sequence and timing of inputs are crucial, such as language modeling, speech recognition, and time-series prediction.
        • Variable-Length Sequences: Capable of handling sequences of variable length, RNNs are adept at processing natural language texts of different sizes or time-series data with varying durations.
      • Limitations and Challenges:
        • Vanishing Gradient Problem: RNNs can struggle with learning long-term dependencies due to gradients diminishing over many time steps, a challenge often mitigated by advanced variants like LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Units).
        • Computational Complexity: Processing sequences step-by-step can lead to increased computational time, especially for long sequences.
      • LSTM, GRU,
  • Large Vision Models (LVMs) Overview
    • The field of computer vision has seen remarkable advancements with the development of Large Vision Models (LVMs). These models leverage extensive datasets and innovative architectures to understand and interpret images in groundbreaking ways.
  • Models
    • CLIP (Contrastive Language-Image Pretraining)
      • Developer: OpenAI
      • Introduction: A vision-language model that understands images in context with natural language, enabling versatile applications such as image captioning, visual question answering, and image retrieval.
      • Capabilities: Facilitates a wide range of image and language tasks, showcasing the power of combining textual and visual information.
    • Google’s Vision Transformer (ViT)
      • Concept: Employs a transformer architecture, traditionally used in NLP, to process image patches, achieving remarkable success in image classification tasks.
      • Achievements: State-of-the-art performance across various benchmarks, highlighting the effectiveness of NLP techniques in visual domains.
    • LandingLens™
      • Developer: LandingAI
      • Introduction: A platform designed to democratize the creation of custom computer vision solutions, requiring no prior coding experience.
      • Features: Enables the creation of tailored vision models for specific industrial applications, such as defect detection, with an emphasis on accessibility and efficiency.
    • ALBEEDO
      • Developer: Facebook AI
      • Introduction: A comprehensive detection model trained on a vast dataset of images and bounding boxes for object detection, image segmentation, and instance segmentation.
      • Capabilities: Designed to understand complex visual scenes and identify multiple objects accurately.
    • DeiT (Data-efficient Image Transformer)
      • Developer: Microsoft
      • Introduction: A subset of vision transformer models focusing on data efficiency, requiring less data to achieve competitive performance.
      • Specialty: Optimized for scenarios with limited data availability, showcasing the adaptability of transformer models to different data scales.

Computer Vision

  • Supervised
    • Supervised Learning Overview:
      • Involves training a model on a labeled dataset with input-output pairs.
      • Aim: Learn a mapping from inputs to outputs to make accurate predictions on unseen data.
      • Iterative training adjusts model parameters (weights, biases, filter values) to minimize prediction errors.
    • Loss Functions in CNNs:
      • Measure the difference between model predictions and true labels.
      • Guide the learning process by providing an objective metric to minimize.
      • Crucial for iterative parameter adjustment and improving model accuracy.
    • Cross-Entropy Loss:
      • Popular for image classification tasks.
      • Measures the dissimilarity between predicted probability distributions and actual distributions.
      • Penalizes incorrect predictions heavily, encouraging the model to increase confidence in correct class predictions.
      • Works with softmax activation in the output layer to output probabilities over classes.
    • Properties and Importance:
      • Logarithmic Nature: Loss increases as the model’s probability for the correct class decreases, and vice versa.
        • Encourages models to be confident in the correct class and cautious with incorrect classes.
        • Essential for achieving high performance in classification tasks by guiding the model towards accurate and confident predictions.
  • Backpropagation
    • Backpropagation in CNNs:
      • Essential algorithm for training CNNs, allowing for gradient calculation and parameter updating.
      • Applies the chain rule of calculus to compute gradients of the loss function with respect to network parameters.
      • Involves multiple steps including forward pass, loss calculation, gradient computation, and parameter updating.
    • Process Overview:
      1. Forward Pass: Input data is processed through convolutional, pooling, and fully connected layers to produce a prediction.
      2. Loss Calculation: The prediction is compared to the true output to compute the loss.
      3. Gradient Computation: Gradients of the loss with respect to each parameter are calculated to understand loss sensitivity.
      4. Backward Pass: Gradients are propagated back through the network using the chain rule, updating parameters to minimize loss.
    • Layer-specific Gradient Propagation:
      • Convolutional Layers: Gradients calculated with respect to both input feature maps and convolutional filters, using a technique similar to the forward convolution operation.
      • Max Pooling Layers: Gradients propagated back to the neuron that produced the maximum value in each window during the forward pass, with other neurons receiving a gradient of zero.
      • Average Pooling Layers: Gradients evenly distributed among all neurons contributing to the window’s pooled output in the forward pass.
      • Fully Connected Layers: Gradients of the loss function computed with respect to weights and biases through the derivative of the neuron’s activation function.
    • Role of Optimization Algorithms:
      • Guided by algorithms like stochastic gradient descent, parameters are iteratively updated using the calculated gradients.
    • Importance:
      • Backpropagation enables neural networks to learn from complex data, adjusting parameters to capture intricate patterns.
      • Works in concert with the network architecture, regularization, and hyperparameters to converge towards optimal solutions.
  • Optimization
    • Gradient Descent Overview:
      • Iteratively updates model parameters using calculated gradients to minimize the loss function.
      • Analogous to finding the lowest point in a mountainous terrain by moving in the direction opposite to the steepest slope.
    • Update Rule: Adjusts parameters by subtracting a fraction (learning rate) of the gradient of the loss function.
    • Gradient Descent Variants:
      1. Batch Gradient Descent:
        • Uses the entire training dataset to compute the gradient for each update.
        • Provides stable gradient estimates but is computationally expensive, especially with large datasets and complex models.
      2. Stochastic Gradient Descent (SGD):
        • Updates parameters using the gradient computed from a single data point or a small batch.
        • Offers faster updates and can avoid local minima but introduces noise that can affect convergence.
      3. Mini-Batch Gradient Descent:
        • Balances between batch and stochastic versions by using a mini-batch of data for computing gradients.
        • Reduces noise and improves computational efficiency; mini-batch sizes often range from tens to hundreds, usually in powers of two.
    • Adaptive Learning Rate Methods:
    • Adam (Adaptive Moment Estimation):
      • Combines the advantages of AdaGrad and RMSProp methods.
      • Adjusts the learning rate for each parameter based on the moving averages of the first and second moments of the gradients.
      • Known for its robust performance across various tasks and is a popular choice for CNN optimization.
    • Key Takeaways:
      • Choosing the right optimization algorithm is critical for effective training, with Adam often being a safe and efficient choice.
      • Optimization techniques balance between accuracy of gradient estimation and computational efficiency.
      • Adaptation in learning rate can lead to more stable and faster convergence.
  • Regularization Techniques:
    • Purpose: Introduce penalties or modifications to encourage models to learn simpler, more general patterns.
    • Dropout: Randomly omits neurons during training, forcing the network to learn redundant representations and preventing over-reliance on specific neurons.
    • L1 and L2 Regularization: Adds penalty terms based on the magnitude of model parameters to the loss function. L1 promotes sparsity, while L2 (weight decay) smoothens learned features.
    • Batch Normalization: Standardizes inputs to a layer across mini-batches, improving training speed and stability with implicit regularization effects.
  • Data Augmentation:
    • Enhances model generalization by artificially increasing training dataset size and diversity through image transformations.
    • Rotation: Rotates images by a random angle within a specified range.
    • Translation: Shifts images horizontally or vertically.
    • Flipping: Horizontally or vertically flips images. Useful when such transformations are realistic (e.g., horizontal flips for car images).
    • Random Cropping: Scales images and takes random crops, effective for creating variability but computationally expensive.
    • Color Transformations: Adjusts image properties like brightness, contrast, and saturation to create varied training examples.
    • Scaling: Uniformly or non-uniformly scales images within a specified range.
  • Impact of Regularization and Data Augmentation:
    • By incorporating these strategies, CNNs learn more robust features that are less likely to overfit to the training data.
    • These techniques improve model performance on unseen data by promoting generalization.
  • CNN
    • Convolutional Layers Overview:
      • Backbone of convolutional neural networks (CNNs).
      • Capture spatial features in images.
    • Key Concepts:
      • Filters/Kernels: Small matrices with trainable weights that detect specific features.
      • Depth Matching: Filter depth should match input data depth (e.g., RGB images require 3-channel filters).
      • Hierarchy of Features: Filters learn complex features over time, forming a hierarchy.
      • Multiple Filters: Allow the capture of various features simultaneously.
      • Receptive Field: The input region contributing to an output feature map. 3x3 convolutions balance receptive field size and computational efficiency.
      • Convolution Operation: Sliding a filter over the input data, performing element-wise multiplication, and summing the results to produce an output feature map.
      • 3D Convolution: Applied to RGB images using 3x3x3 filters to capture patterns across spatial and depth dimensions.
    • Convolution Operation Details:
      • Stride: The number of pixels the filter moves across the input. A stride of 1 results in a densely computed feature map, while larger strides reduce spatial dimensions.
      • Padding:
        • No Padding: Does not add extra pixels around the input, leading to smaller output volumes.
        • Same Padding: Adds extra pixels to maintain the input volume’s spatial dimensions in the output.
    • Output Volume Size Factors:
      • Influenced by input volume size, filter size, stride, and padding type.
    • Efficiency and Feature Capture:
      • Stacking 3x3 convolutional layers can capture the same information as a larger single layer with fewer parameters.
      • Padding types (no padding vs. same padding) affect the output volume size and detail preservation.
    • Types of Convolutions:
      • Standard (2D) Convolution:
        • Slides a 3D filter through an input layer, capturing spatial features like edges and textures.
      • Transposed Convolutional Layers:**
        • They are used for upsampling the spatial dimensions of input feature maps, making them larger.
        • Essential in tasks that require generating or reconstructing images from condensed representations, such as in the decoder part of autoencoders or the generator part of GANs.
        • Deconvolutional
      • Depthwise Convolution:
        • Convolves each input channel separately, reducing computational complexity and parameters.
        • Suitable for efficient spatial feature capturing.
      • Pointwise (1x1) Convolution:
        • Changes the depth of feature maps without affecting spatial dimensions.
        • Used for dimensionality reduction and channel combination.
      • Depthwise Separable Convolution:
        • Combines depthwise and pointwise convolutions.
        • Captures spatial information and increases output feature map depth efficiently.
      • Dilated (Atrous) Convolution:
        • Introduces a dilation factor to expand the filter’s receptive field.
        • Useful for capturing larger patterns and context without increasing size or parameters.
      • Transposed Convolution:
        • Upsamples or increases spatial dimensions of input data.
        • Common in image segmentation and generative models for producing higher-resolution outputs.
      • Efficiency and Application:
        • Depthwise and Pointwise Convolutions: Reduce computational load while maintaining feature extraction capabilities.
        • Depthwise Separable Convolutions: Offer lightweight models with good performance, ideal for mobile applications.
        • Dilated Convolutions: Enhance model’s ability to understand larger contexts and detailed patterns.
        • Transposed Convolutions: Essential for tasks requiring detailed, high-resolution outputs from lower-resolution inputs.
      • other
        • flattened
        • spatial
        • cross spatial
        • shuffled group convolutions
  • Pooling
    • Pooling Layers Overview:
      • Reduces spatial dimensions of feature maps.
      • Retains important information while reducing parameters and computations.
      • Adds spatial invariance, enhancing generalization.
      • Works alongside convolutional layers for hierarchical feature learning.
    • Types of Pooling Layers:
      • Max Pooling:
        • Utilizes a small window (e.g., 2x2, 3x3) to slide over the input and selects the maximum value.
        • Retains prominent features, providing robustness to small input variations.
      • Average Pooling:
        • Calculates the average of values within the window, considering both prominent and less prominent features.
        • Results in a smoother feature map representation.
      • Global Average Pooling:
        • Reduces each feature map to a single value by calculating the average of all values.
        • Useful for capturing global context and reducing dimensions before classification layers.
      • Adaptive Pooling:
        • Adjusts pooling window size based on input dimensions and desired output size.
        • Ensures efficient pooling across varying input sizes, optimizing for different feature map dimensions.
    • Contribution to CNNs:
      • Pooling layers play a crucial role in learning hierarchical features, reducing computational complexity, and improving model generalization.
      • They enable CNNs to handle input data variations and reduce overfitting by summarizing the presence of features.
  • Activation
    • Activation Functions Overview:
      • Introduce non-linearity, allowing networks to model complex relationships.
      • Without them, networks could only model linear relationships.
    • Common Activation Functions:
      • ReLU (Rectified Linear Unit):
        • Formula: (f(x) = \max(0, x))
        • Clips negative values to 0, retains positive values.
        • Efficient, helps reduce overfitting and mitigates vanishing gradient problem.
        • Can suffer from “dead neurons” issue.
      • Sigmoid:
        • Formula: (f(x) = \frac{1}{1 + e^{-x}})
        • Maps inputs to a range between 0 and 1.
        • Useful for probabilities/binary outputs.
        • Smooth but can suffer from vanishing gradients.
      • Hyperbolic Tangent (tanh):
        • Maps inputs to a range between -1 and 1.
        • Zero-centered output, smooth and differentiable.
        • Similar issues with vanishing gradients but generally better than sigmoid.
      • GELU (Gaussian Error Linear Unit):
        • Inspired by the Gaussian error function.
        • Similar to ReLU but non-zero for negative inputs.
        • Performs well in deep learning, especially transformers.
      • SiLU (Sigmoid Linear Unit) / Swish:
        • Formula: (f(x) = x \cdot \text{sigmoid}(x))
        • Smooth, differentiable, combines sigmoid and linear properties.
        • Outperforms ReLU by adapting linear and non-linear behavior.
      • ReLU6:
        • Variant of ReLU, clips output at 6.
        • Introduces bounded non-linearity, useful in quantized networks.
      • Hard-Swish Activation Function:
        • Employed a piece-wise linear approximation of the swish activation, balancing computational efficiency and expressive power. (MobileNet V3)
    • Choosing Activation Functions:
      • Depends on problem, network architecture, and trade-offs (efficiency, robustness, gradient issues).
      • Experimentation is key to finding the most suitable function for a specific use case.
  • FC
    • Fully Connected Layers in CNNs:
      • Role: Map high-level features extracted by previous layers to the desired output (e.g., class labels for image classification).
      • Function: Interpret and combine learned features into a probability distribution over target classes.
    • Process:
      1. Flattening: Prepares multi-dimensional feature maps for FC layers by converting them into a single, one-dimensional vector.
      2. Weight Matrix and Bias Vector:
        • Each FC layer has a weight matrix (W) and a bias vector (b).
        • The weight matrix dimensions are (n \times m), where (n) is the number of neurons in the layer, and (m) is the length of the flattened vector.
        • The bias vector has dimensions equal to the number of neurons in the layer.
      3. Transformation: The input vector is multiplied by the weight matrix and added to the bias vector, producing a new vector representing the weighted sum of the inputs.
      4. Activation Function: Applied element-wise to introduce non-linearity, allowing the network to learn complex input-output relationships.
    • Final Layer:
      • The last FC layer is crucial for generating predictions, with neurons equal to the number of classes.
    • Softmax Activation: Commonly used in the final layer for multi-class classification, converting outputs into a probability distribution over classes.
    • Importance:
      • FC layers weigh the importance of different features for final predictions, handling complex, non-linear relationships.
      • Serve as the interpretive step in CNNs, bridging feature extraction and output generation.
  • Transfer Learning
    • Introduction
      • Challenge: Gathering enough labeled data, computational resources, and time for new deep learning models.
      • Solution: Use of transfer learning to save time and resources.
      • Technique:
        • Pre-train a model on one task, then fine-tune it for another related task.
      • Benefits of Transfer Learning:
        • Saves time and computational resources.
        • Initial layers learn low-level features (edges, textures) useful for new tasks.
        • Reduces training requirements on large labeled data sets.
        • Leads to better generalization from pre-training on related tasks.
        • Faster training and less resource-intensive than starting from scratch.
        • Pre-trained models (often on ImageNet) bring knowledge that improves accuracy and performance on new tasks.
        • Requires substantially less labeled data for training.
        • More likely to generalize well to new, unseen data.
      • Conclusion:
        • Transfer learning enables more efficient problem-solving with less data, less computation, and less time.
    • When to Use Transfer Learning:
      • When
        • Effective for smaller or less diverse datasets than the pre-trained model’s training dataset.
        • Suitable when dataset features are similar to those in the pre-training dataset.
        • For large datasets significantly different from the pre-trained model’s, consider training from scratch.
      • Choosing Between Feature Extraction and Fine-Tuning:
        • Based on dataset size and similarity to the pre-trained model’s task.
        • Feature extraction for smaller datasets to reduce overfitting.
        • Fine-tuning for larger datasets or to adjust pre-trained features to new tasks.
      • Scenario-based Layer Freezing Strategy:
        1. Small & Similar Dataset: Freeze weights up to the last layer, replace the fully connected layer, retrain.
        2. Large & Similar Dataset: Freeze early layers, retrain later layers with a new fully connected layer.
        3. Small & Different Dataset: Remove near-output convolutional layers, retrain closer-to-input layers.
        4. Large & Different Dataset: Retrain the entire network from scratch, replace the fully connected output layer.
      • Fine-Tuning Considerations:
        • Use a lower learning rate to avoid drastic changes to pre-trained weights.
        • Apply data augmentation techniques for smaller datasets to help the model generalize better.
      • General Tips:
        • Understand your dataset size and similarity to the pre-training dataset.
        • Leverage feature extraction for smaller, similar datasets.
        • Use fine-tuning for larger datasets similar to the original model’s task.
        • Freeze layers appropriately to prevent overfitting.
        • Adjust freezing strategy based on dataset size and similarity.
        • Enhance model performance with a lower learning rate and data augmentation.
    • Types
      • Pre-trained Models:
        • Serve as a starting point for new models.
        • Typically trained on large datasets like ImageNet.
        • Help overcome data availability and computational challenges.
        • When to Use Feature Extraction vs. Fine Tuning:
          • Dataset Size: Feature extraction for small datasets to avoid overfitting; fine-tuning for larger datasets.
          • Dataset Similarity: Feature extraction if new dataset is similar to pre-training dataset; fine-tuning if domains are different.
          • Computational Resources: Feature extraction is less resource-intensive and quicker; consider it if resources or time are limited.
      • Feature Extraction:
        • Leverages learned features of pre-trained models.
        • Involves removing the last few layers and adding a new classification layer.
        • Weights of the pre-trained model are kept frozen.
        • Used when dataset is small but similar to the original pre-trained model’s dataset.
        • Feature Extraction Steps:
          1. Select a Pre-trained Model: Choose one pre-trained on a large, diverse dataset (e.g., ResNet, MobileNet, EfficientNet on ImageNet).
          2. Remove Top Layers: Modify the pre-trained model by removing its top classification layers.
          3. Add New Output Layer: Customize by adding a new output layer that matches your use case’s class count.
          4. Freeze Pre-trained Layers: Make the layers non-trainable to prevent updating during new dataset training.
          5. Train the Model: Use your dataset to train only the new model’s layers.
      • Fine Tuning:
        • Adjusts pre-trained model’s weights for the new task.
        • More extensive modifications than feature extraction.
        • Some layers are unfrozen to update weights during backpropagation.
        • Requires more data and resources but adapts better to new tasks.
        • The choice between feature extraction and fine tuning depends on:
        • Size and similarity of the new dataset to the original dataset.
        • Available computational resources.
        • Fine Tuning Steps:
          1. Choose and Modify Pre-trained Model: Start with selecting a pre-trained model and replace the output layer for your task.
          2. (Optional) Freeze and Train New Output Layer: Initially, you might freeze pre-trained layers to train a new output layer, improving performance.
          3. Unfreeze Some Pre-trained Layers: Update weights by unfreezing, especially later layers for task-specific features.
          4. Set Lower Learning Rate: Use a smaller learning rate to adjust pre-trained weights carefully, as the model is already close to a solution.
          5. Train the Model: Update weights of both new and some/all unfrozen pre-trained layers with the new dataset.
  • Main Models
    • LeNet
      • Overview of LeNet:
        • Purpose: Introduced to efficiently extract hierarchical features from input images for tasks such as digit classification on the MNIST dataset.
        • Impact: Paved the way for the development of more advanced deep learning-based computer vision techniques by showcasing the ability to learn features directly from raw data.
      • Architecture Highlights:
        • LeNet’s design is foundational, yet relatively simple compared to today’s CNN architectures.
        • It consists of two main parts: the feature extraction layers (convolutional and pooling layers) and the classification layers (fully connected layers).
      • Detailed Architecture:
        1. Input Layer: Accepts grayscale images of size 32x32 pixels.
        2. First Convolutional Layer: Uses six 5x5 convolutional kernels (filters), resulting in six feature maps of size 28x28, followed by a hyperbolic tangent activation function.
        3. First Pooling Layer: Applies 2x2 average pooling with a stride of 2, reducing the feature maps to 14x14, followed by a hyperbolic tangent activation.
        4. Second Convolutional Layer: Comprises 16 5x5 convolutional kernels, producing 16 feature maps of size 10x10, each passed through a hyperbolic tangent activation.
        5. Second Pooling Layer: Similar to the first pooling layer, it further reduces the feature maps to 5x5.
        6. Third Convolutional Layer: This layer functions similarly to a fully connected layer, applying a 5x5 kernel to each of the 16 input feature maps, resulting in 120 output feature maps of size 1, followed by a hyperbolic tangent activation.
        7. Fully Connected Layers: Bridges the gap between feature extraction and classification, ending with an output layer that generates a probability distribution over class labels.
      • Innovations and Contributions:
        • Feature Learning: LeNet demonstrated that CNNs could automatically learn features from raw data, moving away from hand-engineered feature extraction.
        • Hierarchical Feature Extraction: It showed how deep architectures could learn increasingly abstract representations of the data.
      • Legacy and Tools:
        • While simplistic by modern standards, LeNet laid the groundwork for the extensive research and development in CNNs that followed.
        • Tools like TensorSpace.js allow for interactive exploration of networks like LeNet, providing insights into how different layers process and transform input data.
    • AlexNet
      • Key Contributions of AlexNet:
        • Deeper and Wider Networks: Introduced an 8-layer deep network, showing that depth and width in neural architectures could significantly improve performance on image classification tasks.
        • ReLU Activation: Employed the Rectified Linear Unit (ReLU) for faster training and addressing the vanishing gradient problem, enabling deeper network training.
        • GPU Computing: Leveraged GPU power for efficient training, setting a precedent for the use of GPUs in deep learning.
        • Data Augmentation: Utilized techniques like random cropping and flipping, enhancing generalization by artificially increasing the dataset’s diversity.
        • Dropout Regularization: Implemented dropout to prevent overfitting, enhancing the model’s ability to generalize to unseen data.
        • Stacked Convolutional Layers: Demonstrated the efficacy of stacking multiple convolutional layers before pooling to capture more complex features.
      • Architecture Overview:
        • Begins with an input layer that accepts raw pixel values.
        • Features a series of convolution and pooling layers to extract low-level features (edges, lines) and reduce spatial dimensions while retaining important features.
        • Stacked convolutional layers capture complex patterns and hierarchies.
        • Three fully connected layers towards the end transform feature maps into predictions.
      • Understanding Feature Maps and Layers:
        • Early layers capture basic features like lines and edges.
        • As the network progresses, layers capture more complex and abstract patterns beyond simple geometric shapes.
        • Fully connected layers compile these complex features into a final prediction, classified by the highest activation value.
      • Legacy and Impact:
        • Although surpassed by newer architectures, AlexNet marked a significant shift towards deeper networks in the field.
        • Its strategies for training, regularization, and architecture design continue to influence the development of deep learning models.
    • VGG
      • VGG Overview:
        • Known for its simplicity and depth, VGG introduced a uniform architecture that significantly influenced modern CNN design.
        • Demonstrated that deeper networks with more layers could achieve better performance on image classification tasks.
      • Key Contributions of VGG:
        • Depth and Uniformity: Employed a consistent use of small 3x3 convolutional filters and multiple stacking layers, enabling the network to learn complex features with fewer parameters.
        • Modular Design: Introduced a modular approach to constructing deeper architectures by repeating blocks of layers, simplifying scaling and generalization.
        • Small Filters: Utilized 3x3 convolutional filters throughout, reducing parameters and computational complexity while capturing local information effectively.
      • Impact on Deep Learning:
        • Inspiration for Deeper Architectures: Encouraged the development of even deeper models, such as ResNet and DenseNet, which introduced novel concepts like skip connections to train very deep networks effectively.
        • Standardization of Filter Sizes: Established the use of small filters as a standard practice, influencing future network designs to adopt 3x3 filters for efficiency and performance.
      • VGG-16 Architecture Breakdown:
        • Input Layer: Accepts raw pixel values of the input image.
        • Convolutional Layers: Comprises 13 convolutional layers using 3x3 filters for feature extraction.
        • Max Pooling Layers: Applied after every two convolutional layers to reduce feature map size, preserving important features through spatial downsampling.
        • Fully Connected Layers: Three layers that transform high-level features into class probabilities for image classification.
      • Legacy:
        • VGG’s introduction of a more standardized and deeper architecture paved the way for exploring and designing CNNs with increased depth and complexity.
        • Its modular design principle is now a foundational concept in deep learning, allowing for flexible network configurations and experiments with different depths.
    • ResNet
      • ResNet Overview:
        • Introduced skip connections (shortcut connections or residual connections) as its core innovation.
        • Designed to train very deep networks by facilitating gradient flow across layers, mitigating the vanishing gradient problem.
        • Demonstrated that networks could get deeper without performance degradation, enabling the training of architectures with 18, 34, 50, 101, and even 152 layers efficiently.
      • Key Features and Principles:
        • Skip Connections: Allow outputs of one layer to be added to another layer deeper in the network, simplifying learning by focusing on residual functions.
        • Bottleneck Layers: Designed to reduce computational complexity through a sequence of 1x1, 3x3, and 1x1 convolutions, optimizing the processing of high-dimensional data.
        • Hierarchical Feature Learning: Employs stages of residual blocks to capture increasingly complex features, with max pooling to abstract spatial information and expand the receptive field.
      • ResNet-50 Architecture:
        • Starts with an initial convolution and max pooling layer to reduce image size while preserving details.
        • Followed by stages of convolution and identity blocks with 1x1, 3x3, and 1x1 convolutions, progressively increasing feature map depth from 128 to 512.
        • Utilizes global average pooling and fully connected layers for classification, leveraging skip connections to enhance gradient flow and learning efficiency.
      • Impact on Deep Learning:
        • Deep Architecture Training: Showed that deeper networks could be trained more effectively, influencing the development of subsequent deep learning models.
        • Innovation in Design: Inspired new architectures that build on ResNet’s principles, such as DenseNet and networks employing skip connections for improved learning dynamics.
        • Widespread Adoption: ResNet’s design choices have become foundational, influencing research and practical applications in image classification, feature extraction, and beyond.
    • MobileNet
      • Version 1
        • MobileNetV1 Overview:
          • Developed to address the constraints of mobile and embedded devices, which include limited computing power, memory, and energy.
          • Key innovations include depthwise separable convolutions and the introduction of width and resolution multipliers to enhance computational efficiency.
        • Depthwise Separable Convolutions:
          • Splits traditional convolutions into depthwise and pointwise (1x1) convolutions, significantly reducing the number of parameters.
          • Depthwise convolution applies a single filter to each input channel.
          • Pointwise convolution combines the depthwise convolution outputs, maintaining model effectiveness with far fewer computations.
        • Width Multiplier:
          • Allows the adjustment of the number of channels in each layer, effectively “thinning” the network to reduce its size and computational demand.
          • Requires careful tuning to balance efficiency and model performance.
        • Resolution Multiplier:
          • Adjusts the input image size to manage computational requirements further, enabling the model to operate on smaller images with preserved essential features.
        • MobileNetV1 Architecture Highlights:
          • Begins with a standard convolution layer with 32 filters (3x3) with stride 2 for initial feature extraction.
          • Utilizes depthwise separable convolutions to capture more complex features efficiently.
          • Incorporates batch normalization and ReLU activation for training stability and non-linearity.
          • Employs global average pooling to reduce feature map dimensions, enhancing computational efficiency.
          • Concludes with fully connected layers and a softmax layer for classification, producing a probability distribution over classes.
        • Impact and Applications:
          • MobileNetV1’s design principles have paved the way for creating efficient neural network architectures suitable for devices with stringent resource limitations.
          • Its innovations allow for deep neural networks to be deployed in a wide range of mobile and embedded device applications, achieving good performance with lower computational costs.
      • Version 2
        • MobileNetV2 Innovations:
          • Inverted Residual Blocks: Features expansion and contraction layers, starting with a smaller number of channels, expanding, then contracting, contrary to traditional residual blocks. This structure optimizes computational efficiency and accuracy.
          • Linear Bottleneck: Utilizes a linear activation function in the bottleneck layers, reducing trainable parameters and preserving important features without adding non-linearity that could degrade information.
          • Depthwise Separable Convolutions: Continues the use of depthwise and pointwise convolutions from MobileNetV1, further reducing parameters and computational demand.
        • Architecture Overview:
          • Begins with a standard convolutional layer followed by batch normalization and ReLU6 activation, setting the stage for feature extraction.
          • The core of MobileNetV2 consists of bottleneck residual blocks that sequentially process the input, learning an increasing number of filters.
          • Uses a global average pooling layer to reduce spatial dimensions efficiently, followed by a 1x1 convolutional layer for final feature processing.
          • Outputs class probabilities through a softmax activation, avoiding fully connected layers to minimize parameter count and model size.
        • Efficiency and Performance:
          • Designed for high accuracy on image classification tasks with significantly fewer parameters and computational costs than its predecessors and competitors.
          • The architecture is particularly suitable for devices with limited computational resources, such as mobile phones and embedded systems.
        • Design and Usability:
          • MobileNetV2’s architecture is often represented as a table in academic papers, detailing each layer’s configuration for clarity.
          • Its modular design, featuring the innovative use of inverted residual blocks and linear bottlenecks, simplifies adapting the network for various depths and complexities.
      • Version 3
        • MobileNetV3 Innovations:
          • Optimized for Mobile Devices: Designed to achieve high accuracy while being resource-efficient for smartphones and IoT devices.
          • Neural Architecture Search (NAS): Utilized NAS in combination with manual engineering to optimize the architecture for performance and efficiency.
          • Squeeze-Excitation (SE) Modules: Introduced channel-wise attention mechanisms to enhance feature representation without significant computational overhead.
          • Hard-Swish Activation Function: Employed a piece-wise linear approximation of the swish activation, balancing computational efficiency and expressive power.
        • Design Principles:
          • Inverted Residual Blocks: Builds upon V2’s approach with enhancements for even greater efficiency and accuracy.
          • Linear Bottleneck: Continues the use of linear bottlenecks to reduce parameters and focus on essential features.
          • Depthwise Separable Convolutions: Maintains the use of depthwise separable convolutions to minimize computational cost.
        • Architecture Highlights:
          • Starts with an initial 3x3 convolution layer, followed by a series of bottleneck residual blocks that form the core of the architecture.
          • Employs SE modules to adaptively recalibrate channel-wise feature responses, focusing on informative channels.
          • Utilizes the hard-swish activation selectively to improve learning capabilities without increasing computational requirements.
          • Concludes with a global average pooling layer and a final convolution layer, avoiding fully connected layers to reduce parameter count.
        • Impact and Applications:
          • Achieved state-of-the-art performance on several computer vision benchmarks while maintaining efficiency.
          • Serves as a backbone for various tasks beyond image classification, including object detection and segmentation.
          • Flexible and versatile, capable of being tailored to specific use cases and applications where computational resources are limited.
    • EfficienNet
      • EfficientNet Overview:
        • Utilizes compound scaling to uniformly scale network dimensions (depth, width, resolution), optimizing balance for improved performance.
        • Includes a family of models (EfficientNet B0 to B7), each progressively larger and more accurate.
        • Built on the inverted residual block structure with depthwise separable convolutions for efficiency.
      • Key Innovations:
        • Compound Scaling: A novel scaling method that uniformly scales all dimensions of the network based on a fixed set of scaling coefficients.
        • MBConv Blocks: Utilizes MobileNet’s inverted residual blocks, incorporating depthwise and pointwise convolutions for feature extraction with fewer parameters.
        • Squeeze-and-Excitation (SE) Modules: Integrates SE modules within MBConv blocks to dynamically recalibrate channel-wise feature responses, enhancing representational power.
        • Swish Activation: Replaces traditional activation functions like ReLU with Swish, a smooth, non-monotonic function that enhances model performance, especially in deeper networks.
        • ReLU6 and Hard-Swish: Employs ReLU6 in earlier versions and hard-swish in later iterations for efficient activation with bounded outputs.
      • Architecture Details:
        • Begins with a stem layer for initial feature extraction, followed by multiple MBConv blocks for deep feature processing.
        • Uses a combination of expansion and projection layers within MBConv blocks to efficiently manipulate channel dimensions while extracting rich features.
        • Incorporates squeeze-and-excitation optimization within MBConv blocks for selective channel emphasis.
        • Concludes with a global average pooling and a fully connected layer for classification, using a Softmax activation for class probability distribution.
      • Impact and Applications:
        • Sets new standards for efficiency and accuracy in image classification, significantly outperforming previous architectures with fewer parameters and lower computational cost.
        • Ideal for deployment in resource-constrained environments, such as mobile devices, due to its scalable nature and optimized performance.
        • Serves as a foundation for further research and development in efficient deep learning model design.
    • GoogLeNet
      • Inception V1 (GoogLeNet):
        • Introduced in 2014, it was a winner of the ILSVRC (ImageNet Large Scale Visual Recognition Challenge).
        • Key Innovations: Introduced the inception module, which performs several convolutions in parallel and concatenates their results. This design allows the model to capture information at various scales efficiently.
        • Use of 1x1 Convolution: Employed 1x1 convolutions to reduce dimensionality and computational cost, acting as feature pooling.
      • Inception V2:
        • Introduced in 2015, focused on improving the efficiency and accuracy of V1.
        • Key Innovations: Incorporated batch normalization to accelerate training, reduce the internal covariate shift, and introduced factorization of convolutions to reduce the number of parameters.
        • Architecture Adjustments: Modified inception modules to factorize convolutions into smaller operations, reducing computational complexity.
      • Inception V3:
        • Further Advances: Continued the evolution of the architecture with further factorization of convolutions to improve efficiency and reduce overfitting.
        • Key Innovations: Introduced RMSProp optimizer, label smoothing, and factorized the larger convolutional kernels into smaller, more efficient ones. Expanded the use of 1x1 convolutions to channel pooling.
        • Auxiliary Classifiers: Utilized auxiliary classifiers to inject gradient earlier in the network to combat the vanishing gradient problem.
      • Inception V4 and Inception-ResNet:
        • Introduced in 2016, these architectures combine the strengths of Inception modules with residual connections.
        • Inception-ResNet: Merges the Inception architecture with the residual connections of ResNet, leading to faster training times and improved accuracy.
        • Inception V4: Improved upon the inception modules to achieve higher accuracy without residual connections.
        • Key Features: Both versions leverage the benefits of inception modules and introduce residual connections for more effective training of deeper networks.
      • Impact and Applications:
        • State-of-the-Art Performance: Each version of Inception pushed the boundaries of accuracy in image classification tasks.
        • Efficiency and Scalability: The Inception series is known for its efficient use of computational resources, making it suitable for both high-end and mobile applications.
        • Widespread Adoption: Inception architectures have been widely adopted for a variety of computer vision tasks beyond image classification, including object detection and segmentation.
    • R-CNN (Regions with Convolutional Neural Networks)
      • Introduced: 2014 by Ross Girshick et al.
      • Features: Combines region proposals with CNN features for object detection. Significantly improved the accuracy of object detection systems.
      • Limitations: Slow inference due to the high number of region proposals to process.
    • Fast R-CNN
      • Introduced: 2015 by Ross Girshick.
      • Improvements: Integrates feature extraction and classification into a single model, speeding up training and detection time.
      • Features: Uses Region of Interest (RoI) pooling to share computations across proposals.
    • Faster R-CNN
      • Introduced: 2015 by Shaoqing Ren et al.
      • Advancements: Introduces Region Proposal Network (RPN) for generating region proposals directly within the network, significantly improving speed and accuracy.
    • Mask R-CNN
      • Introduced: 2017 by Kaiming He et al.
      • Features: Extends Faster R-CNN by adding a branch for predicting segmentation masks on each Region of Interest, enabling instance segmentation.
      • Achievements: State-of-the-art performance in instance segmentation tasks.
    • SSD (Single Shot MultiBox Detector)
      • Introduced: 2016 by Wei Liu et al.
      • Features: Utilizes a single deep neural network for object detection, eliminating the need for proposal generation and subsequent refinement.
      • Strengths: Offers a good balance between speed and accuracy, suitable for real-time applications.
    • GANs (Generative Adversarial Networks)
      • Introduced: 2014 by Ian Goodfellow et al.
      • Concept: Consists of two networks, a generator and a discriminator, trained simultaneously through adversarial processes to generate new data samples.
      • Applications: Image generation, style transfer, data augmentation, and more.
    • Transformer Models
      • Introduced: 2017 by Vaswani et al. in “Attention is All You Need”.
      • Key Features: Based on self-attention mechanisms, transformers have revolutionized natural language processing and are increasingly applied to computer vision tasks.
      • Impact: Led to the development of models like BERT, GPT series, and Vision Transformer (ViT).
    • Vision Transformer (ViT)
      • Introduced: 2020 by Alexey Dosovitskiy et al.
      • Features: Adapts the transformer architecture for computer vision tasks by treating images as sequences of patches.
      • Impact: Demonstrates that transformer models can achieve state-of-the-art results in image classification without convolutional layers.
    • YOLO
      • YOLO Versions Overview YOLO has been a groundbreaking series in object detection, known for its speed and accuracy. Here’s how it evolved from version 1 to version 9:
      • YOLOv1
        • Introduced: 2015 by Joseph Redmon et al.
        • Features: First to predict bounding boxes and class probabilities with a single network.
        • Limitations: Poor on small objects, localization.
      • YOLOv2 (YOLO9000)
        • Improvements: Anchor boxes, batch normalization, higher resolution images.
        • Performance: Faster, more accurate, detects over 9000 object classes.
      • YOLOv3
        • Enhancements: Three scale detection, logistic regression for objectness, Darknet-53 backbone.
        • Strengths: Better small object detection.
      • YOLOv4
        • Developers: Alexey Bochkovskiy et al.
        • Advancements: Mish activation, CIOU loss, cross mini-batch normalization.
        • Framework: Darknet-based, improved accuracy and speed.
      • YOLOv5
        • Controversy: Released by Ultralytics, naming debated.
        • Features: PyTorch implementation, pre-trained models, easy deployment.
      • YOLOv6
        • Release: By Meituan in 2022, balancing speed and accuracy for edge devices.
        • Characteristics: Not universally acknowledged due to fragmented development.
      • YOLOv7
        • Details: Improves upon YOLOv5 and YOLOv4, known for robustness and efficiency.
        • Development: Versatile across different environments.
      • YOLOv8
      • YOLOv9
      • YOLOv10
      • General Notes
        • Evolution: YOLO series aims at better detection speed, accuracy, and model simplicity across versions.
        • Community and Forks: Development becomes community-driven with various forks offering improvements.
        • Application: Widely used in real-time detection for surveillance, automotive, and consumer applications.
    • Detectron 2
      • Detectron2 Overview:
        • A comprehensive framework for object detection and segmentation tasks that incorporate a wide array of models, including Faster R-CNN, Mask R-CNN, and RetinaNet.
        • Built on PyTorch, it allows for direct manipulation of PyTorch models and easy customization of algorithms.
        • Supports a wide range of tasks such as instance segmentation, panoptic segmentation, and person keypoint detection.
      • Key Features:
        • Modular Design: Designed with a modular architecture, making it flexible for researchers to experiment with different components and configurations.
        • Extensibility: Easily extendable to include new algorithms and models, allowing researchers and developers to contribute to the framework.
        • Performance: Provides high-quality, high-performance implementations of several state-of-the-art algorithms.
        • Training and Inference: Includes tools and features for efficient training on custom datasets and inference on images and videos.
        • Model Zoo: Features a comprehensive model zoo containing pre-trained models on various datasets, facilitating quick experimentation and deployment.
      • Technical Highlights:
        • Utilizes the PyTorch deep learning framework as its backbone, leveraging PyTorch’s flexibility and ease of use.
        • Incorporates advanced features such as multi-GPU training, mixed-precision training, and model quantization for efficient computation.
        • Supports end-to-end training and evaluation of complex models with multiple components, such as backbones, region proposal networks, and segmentation heads.
      • Applications and Impact:
        • Widely used in academia and industry for developing and deploying cutting-edge computer vision algorithms.
        • Facilitates research by providing a high-quality, well-documented codebase for object detection and segmentation.
        • Detectron2’s flexibility and performance have made it a popular choice for applications ranging from autonomous vehicles to content analysis in social media.
    • Models Inspired by LLMs for Computer Vision
      • Vision Transformer (ViT)
        • Introduction: 2020 by Alexey Dosovitskiy et al.
        • Concept: Adapts the transformer architecture to computer vision by treating images as sequences of patches, similar to how words are treated in NLP.
        • Significance: Demonstrated that pure transformer models could achieve state-of-the-art results on image classification tasks, challenging the dominance of CNNs in the field.
      • DETR (DEtection TRansformer)
        • Introduction: 2020 by Facebook AI Research (FAIR).
        • Concept: Utilizes the transformer model for object detection, framing detection as a direct set prediction problem.
        • Features: Eliminates the need for many hand-designed components like non-maximum suppression and anchor generation, simplifying the object detection pipeline.
      • Swin Transformer
        • Introduction: 2021 by Ze Liu et al.
        • Concept: Proposes a hierarchical transformer whose representation is computed with Shifted Windows, enhancing the efficiency and scalability of transformers for dense prediction tasks in computer vision.
        • Applications: Has been successfully applied to a variety of tasks including image classification, object detection, and semantic segmentation.
      • Perceiver and Perceiver IO
        • Introduction: 2021 by DeepMind.
        • Concept: Introduces a general architecture that can handle multiple types of inputs (not just vision), using a transformer-based approach to process data across a wide range of modalities.
        • Features: Capable of handling arbitrarily large inputs by maintaining a constant latent array to interact with input elements through attention mechanisms, making it versatile for various tasks beyond vision.
      • BEiT (Bidirectional Encoder Representations from Image Transformers)
        • Introduction: 2021 by Microsoft.
        • Concept: Inspired by BERT’s approach in NLP, BEiT trains vision transformers by predicting masked patches of images, applying the concept of pre-training and fine-tuning to visual data.
        • Impact: Demonstrates strong performance on image classification and semantic segmentation tasks, further bridging the techniques between NLP and computer vision.
      • DINOv2 (Distilled-INspired NOisy self-supervised learning)
      • BLIP-2 (Bootstrapped Language-Image Pre-training)

Reinforcement Learning

  • RL
    • Understanding Reinforcement Learning:
      • Definition: RL is a type of machine learning where an agent learns to make decisions by performing actions in an environment and receiving rewards or penalties in return. The goal is to learn a policy that maximizes some notion of long-term reward.
    • Key Components: Agent, Environment, Action, Reward, Policy, and Value function.
    • Next Steps in Learning:
      1. Expand Machine Learning Knowledge:
        • Explore supervised learning (learning with labeled data) and unsupervised learning (learning from unlabeled data).
        • Dive deeper into neural networks, understanding their architecture and how they learn from data.
      2. Deep Dive into Deep Learning:
        • Understand the fundamentals of deep learning, which involves training deep neural networks capable of learning very complex patterns.
        • Apply deep learning in various domains, including natural language processing and computer vision.
      3. Advance to Deep Reinforcement Learning:
        • Once comfortable with RL and deep learning separately, progress to deep reinforcement learning. This field combines neural networks with RL principles to solve complex decision-making tasks.
    • Resources for Further Learning:
      • Check out online platforms like LinkedIn Learning for courses on ML, deep learning, and more.
      • Explore real-world RL projects and tutorials at khaulat.github.io for practical insights.
    • Community and Support:
      • Engage with ML communities online to ask questions, share insights, and connect with peers.
      • Follow and connect with experts in the field on professional networks like LinkedIn.
    • Project Suggestions:
      • Try implementing simple RL algorithms like Q-learning on basic environments such as GridWorld.
      • Gradually increase complexity by tackling environments provided by OpenAI Gym, focusing on domains that interest you.
  • DRL
    • Transition from RL to DRL:
      • Scalability Challenge in RL:
        • Traditional RL uses a Q-table to store the value of actions in specific states, which becomes impractical in environments with large or continuous state spaces due to the explosion in size and complexity.
      • Deep Learning Solution:
        • DRL employs deep neural networks to approximate the Q-value function, effectively handling complex, high-dimensional state spaces without needing a discrete, exhaustive Q-table.
    • How DRL Works:
      • Neural Network as Function Approximator: The neural network in DRL takes the environment’s state (and potentially the action) as input and outputs the predicted Q-values for all possible actions, facilitating the selection of the best action without exhaustive search or trial and error.
      • Learning to Predict Rewards: Through interaction with the environment, DRL updates the neural network parameters to accurately predict the future rewards associated with actions, leveraging backpropagation and optimization techniques from deep learning.
    • Advantages of DRL:
      • Efficiency in Large Spaces: Can efficiently process and make decisions in environments with large or continuous state and action spaces, which are challenging for traditional RL.
      • Generalization: Neural networks can generalize from observed states to unseen states, enabling the agent to perform well in complex environments with dynamic changes.
      • Handling Complex Patterns: Deep learning’s capability to extract and learn from high-level features in data allows DRL agents to identify subtle patterns and strategies for decision-making.
    • Applications and Impact:
      • Game Playing: DRL has achieved superhuman performance in complex games like Go, Chess, and various video games, showcasing its decision-making capabilities.
      • Robotics: Used in robotics for tasks requiring nuanced interactions with the environment, such as grasping objects and autonomous navigation.
      • Other Domains: Applicable in areas like automated trading, autonomous vehicles, and personalized content recommendation, where decision-making in complex environments is crucial.
    • Getting Started with DRL:
      • Pre-requisites: Familiarity with reinforcement learning concepts, deep learning models, and neural network training techniques.
      • Learning Resources: Explore DRL through online courses, tutorials, and hands-on projects using frameworks like TensorFlow and PyTorch.
  • Multi-Agent Reinforcement Learning
    • Understanding MARL:
      • Definition: MARL studies how multiple agents learn to make decisions through trial and error, not in isolation but by interacting with each other and the environment.
    • Types of Interactions: Agents in MARL settings can be:
      • Cooperative: Agents work together towards a common goal.
      • Competitive: Agents have opposing goals, often in adversarial settings.
      • Mixed: Agents exhibit both cooperative and competitive behaviors.
    • Challenges in MARL:
      • Non-Stationarity: The environment’s dynamics change not only due to the external factors but also because of the evolving policies of other agents.
      • Credit Assignment: In cooperative tasks, it’s challenging to attribute success to individual agents’ actions, especially in complex scenarios.
      • Scalability: Increasing the number of agents exponentially increases the complexity of the interaction dynamics.
    • Applications of MARL:
      • Team Sports Simulation: Training teams of agents to play sports, like football, requires cooperative strategies within teams and competitive strategies between teams.
      • Autonomous Vehicles: Developing driving policies for autonomous vehicles that navigate and interact safely and efficiently with human drivers and other autonomous vehicles.
      • Robotic Swarms: Coordinating groups of robots to perform tasks collectively, such as search and rescue operations, environmental monitoring, or agricultural tasks.
    • Designing MARL Systems:
      • Environment Setup: Define how agents perceive their environment, the actions they can take, and the rewards they receive for different interactions.
      • Learning Algorithms: Adapt reinforcement learning algorithms to accommodate multiple agents, considering the potential for policy gradient methods, value-based methods, and model-based approaches.
      • Evaluation: Develop metrics and simulation environments to assess the performance, efficiency, and safety of MARL systems in diverse scenarios.
    • Advancements and Research:
      • Ongoing research in MARL explores new architectures, learning algorithms, and theoretical frameworks to address its inherent challenges, aiming to enhance learning efficiency, cooperation, and competitive strategies among agents.
  • Inverse Reinforcement Learning
    • Inverse Reinforcement Learning Overview:
      • Definition: IRL is the process of deriving the reward function based on the observed behavior of an agent operating under an unknown reward function. Essentially, it works backward from observed behaviors to infer the rewards.
      • Objective: The primary goal of IRL is to learn the underlying motivations, objectives, or “rewards” that guide the observed agent’s decisions, rather than learning to mimic the actions directly.
    • Imitation Learning Context:
      • IRL falls under the broader category of imitation learning, where the aim is to learn from demonstrations. IRL differentiates itself by focusing on learning the reward function, which can then be used to guide reinforcement learning agents.
    • Applications:
      • Self-driving Vehicles: IRL can be used to understand human driving behaviors and preferences by observing human drivers, aiming to replicate these behaviors in autonomous driving algorithms.
      • Robotics: In tasks where explicit programming is challenging, robots can learn desired behaviors by inferring the rewards from human demonstrations.
      • Game AI: Developing more human-like AI characters by learning the reward structures behind human player strategies and decisions.
    • Challenges in IRL:
      • Ambiguity: Multiple reward functions can explain observed behaviors, making it challenging to identify the “correct” one.
      • Complexity: The process of deriving reward functions from high-dimensional, continuous action spaces can be computationally intensive.
      • Evaluation: Assessing the accuracy of the inferred reward function is non-trivial, as the true reward function is unknown.
    • Learning and Research:
      • Foundational Papers: The “Survey of Inverse Reinforcement Learning” provides insights into the challenges, methodologies, and advancements in IRL, offering a comprehensive overview for those interested in deepening their understanding.
      • Techniques and Models: Research in IRL continues to evolve, with methodologies leveraging deep learning, probabilistic models, and Bayesian inference to better infer reward functions from complex behaviors.

Code

  • Pytorch
    • a
  • TF
    • ModelCheckpoint
      • ModelCheckpoint Overview:
        • Purpose: Saves the model at specified intervals (e.g., after every epoch) during training, allowing for the recovery of model states that offer the best validation performance.
        • Key Benefit: Ensures the last saved model reflects the best version based on monitored metrics, such as validation accuracy.
      • Implementation Steps:
        1. Set Up Checkpoint Directory:
          • Define a directory where the model checkpoints will be saved.
        2. Configure ModelCheckpoint:
          • Specify the filepath for saving the model.
          • Choose a metric to monitor (e.g., validation accuracy).
          • Decide whether to save only the best model (save_best_only=True) or save the model after every epoch regardless of performance.
        3. Integrate with Model Training:
          • Add the configured ModelCheckpoint instance to the model’s training process via the callbacks parameter.
      • Code Example: ```python from tensorflow.keras.callbacks import ModelCheckpoint checkpoint_dir = ‘./checkpoints/’ filepath = checkpoint_dir + “model-{epoch:02d}-{val_accuracy:.2f}.hdf5” model_checkpoint_callback = ModelCheckpoint( filepath=filepath, monitor=’val_accuracy’, verbose=1, save_best_only=True, mode=’max’ ) model.fit(x_train, y_train, validation_data=(x_val, y_val), epochs=10, callbacks=[model_checkpoint_callback])