Academic Databases
Explore Our Semantic Databases
STEM & Tech
Math, Physics, CS & Engineering...
Economics
Macro, Micro, Econometrics...
Business
Management, Marketing, Finance...
Health & Life Sciences
Biology, Medicine, Neuroscience...
Humanities & Social Sciences
Psychology, Law, Philosophy, History...
General Studies
General & interdisciplinary studies...
Explore Topics
Features

Deploying YOLOv8 on NVIDIA Jetson Edge Devices: A Practical Guide

Added: 2026-08-07

Deep Dive

Fundamental concepts of Object Detection, specifically the architecture and evolution of the YOLO (You Only Look Once) model family.

You Only Look Once: object detection with YOLOv4 | DIY drone pt. 3

You Only Look Once: object detection with YOLOv4 | DIY drone pt. 3

11K

YOLO (You Only Look Once) is a neural network family that detects objects in images at incredibly fast speeds by dividing images into regions and processing them in parallel. Introduced in 2016 by Joseph Redmon, it achieved a hundred times speed boost over R-CNN variants. The evolution continued with YOLO 9000 (2017), YOLO v3 (2018) which improved small object detection, and YOLO v4 (2020). Despite the original author leaving the field, YOLO v5 emerged as a Python-reimplemented branch. This rapid development demonstrates how open-source computer vision frameworks continuously improve through community contributions.

Click to watch →

[ITA] YOLO: corso completo per principianti! (You Only Look Once)

[ITA] YOLO: corso completo per principianti! (You Only Look Once)

1.9K

YOLO (You Only Look Once) is a revolutionary object detection algorithm that became the state-of-the-art in computer vision. It is not a single model but a family of models starting from 2015 with Joseph Redmon's original paper. The first three versions (YOLO, v2/YOLO 9000, v3) were developed by Redmon's group, which later left the field due to concerns about military applications. The project continued with multiple contributors, including Ultralytics who developed v5, v8, and v11. Object detection combines classification (assigning labels) and localization (finding bounding boxes), unlike earlier two-step approaches like RCNN. YOLO's key advantage is its speed and efficiency, enabling deployment on resource-constrained devices like embedded systems and robots. The model processes images by dividing them into grid cells, predicting confidence scores and bounding boxes in a single pass. Anchor boxes (introduced in v2) help handle objects of various sizes and aspect ratios. Non-Maximum Suppression eliminates redundant boxes, while metrics like Precision, Recall, and Mean Average Precision (mAP) evaluate model performance.

Click to watch →

YOLO - Explained!

YOLO - Explained!

1.7K

This comprehensive section covers the foundational concepts of YOLO. It begins by defining YOLO as an end-to-end trainable neural network for object detection that produces all bounding box predictions and classifications in a single forward pass. The video traces the evolution of object detection architectures: RCNN (50 seconds/inference, 58% mAP), Fast RCNN (2.3 seconds), Faster RCNN (200ms), and YOLO (10x faster than Faster RCNN). Four key advantages are explained: real-time inference, simpler architecture, global context utilization, and learning generalized representations. The neural architecture is detailed: processing 448×448×3 inputs through convolutional layers to produce the 7×7×30 output tensor, where each grid cell contains 20 class probabilities and 10 bounding box predictions (each with x, y, w, h and confidence).

Click to watch →

Лекция. Детекция -- модели, часть 2

Лекция. Детекция -- модели, часть 2

1.2K

The YOLO (You Only Look Once) family represents the most extensive object detection model family, with over 11 main versions developed by multiple research teams since 2015. The evolution from YOLOv3 to YOLOv12 demonstrates progressive architectural improvements: YOLOv3 introduced anchor-free prediction with exponential normalization; YOLOv4 added SPP blocks for multi-scale processing; YOLOv5 revolutionized deployment with an accessible ecosystem; YOLOv6 introduced Task Alignment Learning for anchor-free detection; YOLOv7 added auxiliary heads for better gradient propagation; YOLOv9 implemented dual-head architecture (one-to-many for training, one-to-one for inference) to eliminate NMS; and YOLOv12 adopted Transformer-based architecture with Flash Attention. Beyond YOLO, CornerNet and CenterNet represent anchor-free approaches predicting corner points or object centers instead of anchor boxes, while rotated bounding box detection addresses scenarios like satellite imagery where objects are oriented at various angles. Advanced label assignment methods including Adaptive Training Sample Selection, Encignment, and Optimal Transport Assignment further improve detection accuracy.

Click to watch →

Mengenal YOLO: Implementasi YOLO untuk Deteksi Objek di Webcam, Gambar, dan Video!

Mengenal YOLO: Implementasi YOLO untuk Deteksi Objek di Webcam, Gambar, dan Video!

1K

YOLO (You Only Look Once) is an object detection algorithm that identifies multiple objects within images, videos, or frames. Unlike classification which assigns a single label to an entire image, YOLO detects and classifies each object individually. Built on Convolutional Neural Networks (CNN), YOLO uses a single-pass processing approach for speed and efficiency. The algorithm divides images into S×S grids (typically 7×7 or 8×8), with each grid cell responsible for detecting objects centered within it. YOLO outputs bounding boxes with labels and confidence scores. Originally introduced in 2015, YOLO has evolved through multiple versions (v1-v11), transitioning from the C++-based Darknet framework to PyTorch for easier implementation.

Click to watch →

Deep Dive

Basic familiarity with Deep Learning frameworks (like PyTorch or TensorFlow) and model serialization formats (such as ONNX).

Everything You Want to Know About ONNX

Everything You Want to Know About ONNX

42.1K

ONNX enables complete cross-framework portability demonstrated through end-to-end workflows. Training a CNN in PyTorch, converting to ONNX using PyTorch's exporter, then converting ONNX to TensorFlow's saved model format enables deployment in TensorFlow-based production environments. The complete pipeline involves: training in PyTorch, converting to ONNX, converting ONNX to TensorFlow, then performing native TensorFlow inference without ONNX runtime dependencies. This demonstrates how ONNX serves as a universal bridge between frameworks, enabling organizations to leverage the best tools for each stage of the ML lifecycle while maintaining model portability across the entire ecosystem.

Click to watch →

ONNX: The PDF Format For Neural Networks

ONNX: The PDF Format For Neural Networks

248

ONNX (Open Neural Network Exchange) is a standardized PDF-like format for neural networks that enables interoperability across different machine learning frameworks. Models trained in PyTorch or TensorFlow can be exported to ONNX format and executed using ONNX Runtime without requiring framework-specific dependencies. The PyTorch export process requires dummy data to infer the computation graph, as PyTorch's dynamic graph needs sample inputs to determine the graph structure. The torch.onnx.export() function requires the model, dummy input data, input/output names, and the dynamo flag for modern export. Consistent naming conventions across frameworks are essential for interoperability.

Click to watch →

How my models work: Loading, optimization and execution

How my models work: Loading, optimization and execution

120

Three major frameworks power AI deployment: PyTorch uses eager execution for intuitive debugging but requires manual optimization; TensorFlow employs static graph compilation for production efficiency with less intuitive debugging. ONNX serves as an agnostic bridge, accepting PyTorch models and optimizing them via TensorRT. Each framework has strengths: PyTorch suits research and rapid prototyping, TensorFlow excels in production environments, and ONNX provides cross-platform compatibility. Selecting the right framework depends on development needs versus deployment requirements, with trade-offs between flexibility, performance, and ecosystem support.

Click to watch →

Lightning Talk: What's New in Torch.onnx - Maanav Dalal & Justin Chu, Microsoft

Lightning Talk: What's New in Torch.onnx - Maanav Dalal & Justin Chu, Microsoft

215

TorchONNX is Microsoft's PyTorch module for converting PyTorch models to ONNX format for production deployment. The new exporter architecture uses torch.export and torch.fx for graph capture, replacing torchscript. Key features include: torchlib library supporting dynamic shapes and ONNX opsets 18-24, ONNX IR for in-memory model manipulation, and support for PyTorch 2 control flow operators (torch.cond, torch.while). Native ONNX operators like attention and rotary embedding map directly to single ONNX ops. Custom operators can be created via torch.onnx.ops symbolic API. The export returns an ONNXProgram object with pattern-based fusion optimizations. Validation tools diagnose export accuracy and pinpoint numeric discrepancies. ONNX models can be deployed across AMD, Intel, Qualcomm, and Nvidia hardware for on-device inference.

Click to watch →

How to Run PyTorch Models in the Browser With ONNX.js

How to Run PyTorch Models in the Browser With ONNX.js

34.3K

Converting PyTorch models to ONNX involves: (1) Importing the trained model; (2) Instantiating and loading trained parameters; (3) Setting evaluation mode to bypass dropout; (4) Creating a dummy input tensor matching expected input shape; (5) Calling torch.onnx.export() with the model, dummy input, output path, and verbose flag. ONNX.js inference requires creating an inference session, loading the model, creating input tensors using ONNX.Tensor class, and awaiting the output. A critical limitation is that ONNX.js doesn't support log_softmax, requiring model modifications to use supported operators like softmax instead.

Click to watch →

Deep Dive

Understanding the core differences between Edge Computing and Cloud Computing, including constraints like latency, bandwidth, and power consumption.

Edge Computing Vs. Cloud Computing - Key Differences to Know | Intellipaat

Edge Computing Vs. Cloud Computing - Key Differences to Know | Intellipaat

17.3K

Edge Computing processes data at or near the source of generation, offering benefits like lower latency, reduced bandwidth requirements, improved model accuracy, broader reach without internet access, and enhanced data sovereignty, making it ideal for real-time applications such as IoT devices, smart cities, healthcare, industrial automation, and retail. In contrast, Cloud Computing provides centralized data storage and processing via the internet, offering advantages including scalability, cost savings, flexibility, disaster recovery capabilities, collaboration features, and automatic software updates, making it suitable for applications requiring high processing power, large storage capacity, and high availability. The choice between these paradigms depends on specific organizational needs, with Edge Computing excelling in scenarios demanding immediate response times and local data handling, while Cloud Computing remains optimal for scalable, centralized data management and complex computational tasks.

Click to watch →

What is Edge Computing | How Does Edge Computing Works | Edge Computing | Intellipaat

What is Edge Computing | How Does Edge Computing Works | Edge Computing | Intellipaat

9.5K

Edge Computing addresses three major network constraints: bandwidth limitations, latency delays, and congestion problems. Bandwidth refers to the finite data capacity of networks, measured in bits per second, where wireless communication imposes stricter restrictions. Latency represents the time delay in data delivery between network sites, which can severely impact real-time applications like autonomous vehicles. Network congestion occurs when the volume of data from billions of connected devices overwhelms internet infrastructure, causing slow transmissions and potential outages. These constraints collectively challenge traditional centralized architectures designed for less demanding data environments. Edge Computing provides an effective solution by decentralizing processing to reduce data movement requirements, enabling faster response times and maintaining operational reliability in time-sensitive applications.

Click to watch →

Edge Computing Explained: Architecture, Benefits & Use Cases for Beginners

Edge Computing Explained: Architecture, Benefits & Use Cases for Beginners

477

Edge computing differs from cloud computing in four key areas: (1) Latency - edge offers milliseconds latency versus cloud's over 100ms due to distance, (2) Location - edge is distributed across multiple locations near devices while cloud is centralized, (3) Connectivity - edge can operate with intermittent connectivity while cloud requires constant connection, and (4) Resources - edge has limited resources optimized for specific workloads while cloud offers unlimited scalable resources.

Click to watch →

What is Edge Computing?? | Difference between Edge Computing and Cloud Computing

What is Edge Computing?? | Difference between Edge Computing and Cloud Computing

280

Edge Computing is a distributed computing framework enabling devices in remote locations to quickly process and act on data at the network edge, either by the device itself or a local server. The network edge represents the connection point between enterprise-owned networks and third-party infrastructure, where data is distributed across wireless land, corporate land, and private data centers. Unlike cloud computing where data requests travel to centralized data centers for processing, Edge Computing analyzes data locally at the edges, providing immediate responses without remote data center permission. This fundamental difference enables faster performance and is critical for applications requiring real-time responses, such as self-driving cars and business operations.

Click to watch →

Diana 31st July2020

Diana 31st July2020

107

Cloud computing offers homogeneous environments under single administration with abundant computing power, efficient networks, and management tools, but raises privacy concerns when sending data to centralized servers. Edge computing provides heterogeneous environments with resource-constrained devices, less reliable networks, and limited processing power, requiring continued reliance on cloud services for intensive tasks while offering reduced transmission delays and better local data access.

Click to watch →

Deep Dive

Familiarity with the Linux terminal, bash scripting, and the basics of GPU acceleration concepts (CUDA and cuDNN).

Linux для начинающих / Урок #1 – Что такое дистрибутивы, Bash Shell и Ubuntu?

Linux для начинающих / Урок #1 – Что такое дистрибутивы, Bash Shell и Ubuntu?

574.4K

Linux distributions (distros) are complete operating systems built on the Linux kernel, including Ubuntu, Mint, Elementary, and Android. Distributions vary from Windows-like interfaces to command-line-only systems. Bash scripting enables automation of system tasks through command files. The course covers installation, graphical interface usage, terminal commands, package management, and Bash script creation for system administration.

Click to watch →

Getting Started with Python Deep Learning for Beginners

Getting Started with Python Deep Learning for Beginners

240.6K

This extensive section covers configuring NVIDIA GPU support for deep learning. It explains why GPU acceleration is critical for performance, how to find compatible CUDA and cuDNN versions for your TensorFlow version, downloading and installing CUDA toolkit, extracting and integrating cuDNN libraries, updating system PATH variables, and verifying GPU installation works. The key principle is that specific TensorFlow versions require specific CUDA/cuDNN versions.

Click to watch →

CUDA vs CPU on Linux — Fix Your GPU Not Working (Full Clean Install)

CUDA vs CPU on Linux — Fix Your GPU Not Working (Full Clean Install)

150

CUDA enables dramatically faster data processing by leveraging GPUs' massive parallel architecture (thousands of CUDA cores) compared to CPUs' limited threading capabilities (dozen threads), allowing thousands of tasks to execute simultaneously rather than sequentially; installing CUDA on Linux requires removing existing drivers, adding the NVIDIA repository, installing the toolkit, and configuring environment variables to enable GPU acceleration for applications like machine learning models.

Click to watch →

Full installation of Cuda and Cudnn with Pytorch for all GPU's in Linux

Full installation of Cuda and Cudnn with Pytorch for all GPU's in Linux

18.2K

This tutorial demonstrates how to properly install CUDA toolkit and cuDNN libraries on Linux systems for GPU-accelerated deep learning applications. The process involves downloading CUDA 11.8 from NVIDIA's developer website, installing it via terminal commands, then installing cuDNN 8.6.0 which is compatible with CUDA 11.8. After installation, users verify the setup by importing PyTorch and checking GPU availability using torch.cuda.device_count(). The video emphasizes that different GPU architectures require specific CUDA versions, and Python packages like PyTorch must match the installed CUDA version for optimal performance in machine learning and deep learning applications.

Click to watch →

install cudnn for cuda toolkit in linux mint and ubantu

install cudnn for cuda toolkit in linux mint and ubantu

6.4K

This video demonstrates the complete process of installing CUDA toolkit and cuDNN libraries on Linux systems, including downloading components, running installation commands with sudo privileges, verifying successful installation through test compilation, and troubleshooting common issues like header file syntax errors by converting quotation marks to angle brackets for proper include statements.

Click to watch →

Edge AI Intro

Edge vs Cloud

Jetson Lineup

Hardware Deep Dive

Model Optimization

Performance Benchmarks

Technical Q&A

Advanced Usage

Final Resources

Edge AI Intro

0:00

Playing Section

1
Introduces the session on YOLO performance at the edge.
2
Lakshanta presents Seed Studio's focus on embedded AI hardware.
3
The partnership between Ultralytics and Seed Studio is established.

Deep Dive

Advanced model optimization techniques, such as INT8 quantization and calibration using TensorRT to maximize inference speed.

Making Computer Vision Models Faster: An Introduction to TensorRT Optimization

Making Computer Vision Models Faster: An Introduction to TensorRT Optimization

106

TensorRT is an SDK for high-performance deep learning inference that optimizes pre-trained neural networks for NVIDIA GPU deployment. Key requirements include a trained model (post-training only), an NVIDIA GPU, and optionally sample images for calibration. Core optimization techniques include layer fusion (combining sequential operations like convolution with ReLU or batch normalization into single kernels to reduce memory access overhead), quantization (reducing precision from FP32 to FP16 or INT8), kernel autotuning, constant folding, dead branch elimination, dynamic tensor memory allocation, and dynamic shape profiles. FP16 quantization typically maintains accuracy while enabling Tensor Core utilization, whereas INT8 quantization requires calibration using sample data to estimate activation distributions and may involve fallback to higher precision for problematic layers.

Click to watch →

NVAITC Webinar: Deploying Models with TensorRT

NVAITC Webinar: Deploying Models with TensorRT

20.8K

TensorRT is NVIDIA's software development kit for high-performance deep learning inference that addresses two key challenges: high performance and platform portability across edge devices, autonomous driving platforms, and data center GPUs. It implements five core optimization technologies: kernel fusion (combining multiple kernels into single executions to reduce overhead), precision calibration (using lower precision during inference to reduce model size and latency), kernel auto-tuning (selecting optimal implementations based on configuration), dynamic tensor memory (allocating memory only for tensor duration), and multi-stream execution (enabling parallel inference for multiple clients). The TRTorch library provides a straightforward conversion pipeline that abstracts these optimizations, allowing developers to convert trained PyTorch models into optimized TensorRT modules with minimal code, achieving significant speedups (1.5-6.4x) while maintaining model accuracy.

Click to watch →

From model weights to API endpoint with TensorRT LLM: Philip Kiely and Pankaj Gupta

From model weights to API endpoint with TensorRT LLM: Philip Kiely and Pankaj Gupta

4.9K

FP8 quantization provides approximately 40% performance improvement over FP16 with minimal accuracy loss. Post-training quantization takes only about 5 minutes. FP8 is supported by H100 and L4 GPUs but not A100s. KV cache quantization is critical because the context phase is compute-bound, and quantizing both weights and KV cache enables maximum tensor core utilization. The E4M3 format provides significant dynamic range compared to INT8, allowing better quantization of KV cache values. Calibration determines appropriate min/max ranges for each layer based on data distribution.

Click to watch →

Нейронные сети: быстрый инференс на GPU с помощью TensorRT / Дмитрий Коробченко (NVIDIA)

Нейронные сети: быстрый инференс на GPU с помощью TensorRT / Дмитрий Коробченко (NVIDIA)

6.6K

TensorRT is an NVIDIA library that accelerates neural network inference on GPUs through three key optimizations: layer fusion (combining multiple operations into single kernels), precision reduction (converting from FP32 to FP16 or INT8), and automatic kernel selection (choosing the fastest implementation for each layer based on hardware). This optimization process can achieve up to 2x speedup compared to standard frameworks like TensorFlow, making it essential for deploying trained models in production environments such as data centers, embedded devices, and autonomous vehicles.

Click to watch →

YOLO11 TensorRT Object Detection on Jetson Nano Orin with 100FPS on Live Webcam with Ultralytics

YOLO11 TensorRT Object Detection on Jetson Nano Orin with 100FPS on Live Webcam with Ultralytics

12.1K

This video demonstrates how to optimize YOLO11 object detection models for real-time inference on NVIDIA Jetson Nano using TensorRT, achieving over 100FPS on live webcam by converting PyTorch models to TensorRT engine files with half-precision (FP16) or integer-8 (INT8) quantization, which provides 4-5x faster inference speed compared to running the original PT model while maintaining acceptable accuracy.

Click to watch →

Deep Dive

Integrating deployed models into end-to-end intelligent video analytics pipelines using NVIDIA DeepStream SDK.

Video Analytics for mere mortals with NVIDIA DeepStream SDK - Yousry Mohamed - NDC Melbourne 2022

Video Analytics for mere mortals with NVIDIA DeepStream SDK - Yousry Mohamed - NDC Melbourne 2022

3.1K

This section demonstrates the complete workflow for building video analytics pipelines using DeepStream SDK. It covers: (1) Remote container development setup using Visual Studio Code's Remote Containers extension with pre-built NVIDIA Docker images; (2) Graph Composer interface for visual pipeline creation through drag-and-drop components including sources, preprocessors, inferencers, trackers, and outputs; (3) Pipeline configuration including parameterization for dynamic adjustments, saving pipeline definitions as version-controllable .ml files; (4) Execution using execute_graph command with platform specifications for virtual machines or Jetson devices; (5) Demonstration of basic object detection pipeline producing bounding boxes for people, cars, bicycles, and road signs; (6) Introduction to cascaded inference architecture for layered processing where subsequent models operate on outputs from previous stages.

Click to watch →

DeepStream SDK — Accelerating Real Time AI Based Video and Image Analytics

DeepStream SDK — Accelerating Real Time AI Based Video and Image Analytics

35.2K

Intelligent Video Analytics (IVA) extracts meaningful insights from video data using AI and computer vision, with over a billion cameras deployed globally across retail, traffic engineering, manufacturing, and logistics. DeepStream SDK provides a unified software stack spanning from application level down to hardware, supporting deployment across all NVIDIA platforms from Jetson Nano to data center GPUs. The framework uses a plug-and-play architecture built on GStreamer with key plugins: nvv4l2h265dec/nvjpegdec for hardware-accelerated decode/encode, nvinfer for AI inference using TensorRT, and nvtracker for object tracking. Performance varies significantly: Jetson Nano 2 handles 8 streams while T4 GPUs process up to 68 concurrent streams using YOLOv4-tiny for object detection.

Click to watch →

Best Practices for Real-time Intelligent Video Analytics • Ekaterina Sirazitdinova • GOTO 2021

Best Practices for Real-time Intelligent Video Analytics • Ekaterina Sirazitdinova • GOTO 2021

106.7K

DeepStream SDK builds video analytics pipelines on GStreamer with hardware-optimized plugins for decoding, encoding, inference, and analytics. The pipeline architecture includes camera/audio input, hardware decoding, image preprocessing, batching, inference (TensorRT/Triton), object tracking, visualization, and RTSP output. Zero memory copy eliminates CPU-GPU data transfer bottlenecks by performing all operations on GPU after initial transfer. Best practices include utilizing GPUs fully during training, enabling mixed precision training, applying pruning and quantization-aware training, using TensorRT for automatic optimization, and implementing zero-memory-copy pipelines with DeepStream for maximum efficiency.

Click to watch →

Implementing Real-time Vision AI Apps Using NVIDIA DeepStream SDK

Implementing Real-time Vision AI Apps Using NVIDIA DeepStream SDK

30.6K

NVIDIA DeepStream SDK provides a comprehensive streaming analytics toolkit for building efficient real-time video and image understanding applications, featuring hardware-accelerated plugins for decoding, encoding, and inference, along with transfer learning capabilities for model optimization and cloud-edge orchestration features like bi-directional messaging and over-the-air model updates to enable scalable deployment of intelligent video analytics across industries.

Click to watch →

Hello, AI World. Meet Jetson Nano

Hello, AI World. Meet Jetson Nano

77.4K

DeepStream represents Nvidia's comprehensive solution for intelligent video analytics at the edge, leveraging Jetson's hardware capabilities for maximum throughput. The SDK processes multiple video streams simultaneously using hardware video encoders/decoders, GPU acceleration, and TensorRT-optimized inference engines. A demonstrated use case processes 8 HD video streams at 30 fps each (over 500 megapixels per second total) with ResNet-based object detection networks. The architecture achieves zero-copy data transfer between processing stages, eliminating unnecessary memory copies that would degrade performance. Applications span network video recorders, smart cameras, and IoT gateway systems. The same DeepStream SDK supports both edge devices and cloud servers, enabling edge-to-cloud architectures where initial processing occurs at the edge and aggregated data is transmitted to the cloud for further analysis without framework rewriting.

Click to watch →

Deep Dive

Implementing MLOps for Edge AI, including remote model deployment, over-the-air (OTA) updates, and edge device fleet monitoring.

AWS re:Invent 2020: MLOps for edge devices with Amazon SageMaker Edge Manager

AWS re:Invent 2020: MLOps for edge devices with Amazon SageMaker Edge Manager

1.6K

Amazon SageMaker Edge Manager is a service that enables ML developers to prepare, deploy, and continuously monitor machine learning models across a fleet of edge devices, addressing challenges of resource constraints, model quality decay, and the need for continuous learning by capturing data samples from deployed models, uploading them to the cloud for labeling and retraining, and using automated monitoring to detect performance degradation and trigger model updates.

Click to watch →

Tiny MLOps: Overview, Tools and Challenges

Tiny MLOps: Overview, Tools and Challenges

471

Monitoring consumes significant power; options include direct cloud connectivity, gateway-based architectures, or mesh networks. Model updates pose bricking risks requiring secure transmission, automatic recovery, and partial updates when supported. Security threats include side-channel attacks extracting models from RAM. Robustness considerations: sensor failure tolerance, network failure handling, graceful degradation during low battery, and hardware redundancy (dual systems for fault tolerance). Edge Impulse provides practical tools: device selection showing latency/memory estimates, EON tuner for automatic model training meeting constraints (look ahead implementation), performance visualization showing time distribution between DSP and model prediction. Deployment APIs allow choosing hardware and libraries. Update workflows keep training in the cloud while enabling over-the-air updates. Fleet-wise A/B testing compares models across different device subsets rather than individual devices due to memory limitations.

Click to watch →

What is Edge MLOps?

What is Edge MLOps?

3.5K

Edge MLOps is the application of MLOps principles to Edge AI deployments, focusing on systematizing and automating machine learning workflows including version control, automation, and governance to ensure repeatability, compliance, and efficient deployment of models to edge devices while addressing challenges like model drift and security requirements.

Click to watch →

Edge AI: Where MLOps Ends and Edge Orchestration Begins | Ask the Expert

Edge AI: Where MLOps Ends and Edge Orchestration Begins | Ask the Expert

103

Edge AI deployment leverages container technology to package machine learning models with their serving engines, enabling efficient distribution and management across distributed edge devices. The container abstraction allows models to be versioned and updated independently while maintaining compatibility with the serving infrastructure, with only the model changes being distributed to edge sites. This approach bridges MLOps (model development and management) with edge orchestration (deployment and monitoring), enabling continuous model improvement through feedback loops that capture real-world inference data for retraining and drift detection.

Click to watch →

TinyML at the Edge: Deploying and Optimizing AI Workloads on Zephyr RTOS - Amandeep Singh, Welzin

TinyML at the Edge: Deploying and Optimizing AI Workloads on Zephyr RTOS - Amandeep Singh, Welzin

205

Zephyr RTOS provides mechanisms for dynamic model management including Linkable Loadable Extensions (LLext) enabling runtime updates without device reflashing, and OTA (Over-the-Air) updates using MCU bootloader technology with dual-slot flash memory architecture. LLext supports hot swapping for iterative model development, allowing state persistence through swap memory before component replacement. OTA updates deliver firmware remotely by downloading new images to secondary slots and switching upon reboot. These capabilities accelerate development cycles while enabling field updates for security patches and feature enhancements in deployed edge devices.

Click to watch →

Deep Dive

Designing and training custom YOLOv8 models on proprietary datasets tailored for specific real-world edge applications.

Train Yolov8 object detection on a custom dataset | Step by step guide | Computer vision tutorial

Train Yolov8 object detection on a custom dataset | Step by step guide | Computer vision tutorial

657.6K

This tutorial demonstrates the complete workflow for training a custom object detector using YOLOv8, covering data collection from sources like Open Images Dataset, annotation using CVAT tool, formatting data into YOLOv8's required structure with images in 'images' directory and labels in 'labels' directory, and training the model either locally via Python or Google Colab using the ultralytics library, followed by testing the model's performance on unseen data.

Click to watch →

Training YOLOv8 on Custom Datasets for Object Detection | Step-by-Step Tutorial

Training YOLOv8 on Custom Datasets for Object Detection | Step-by-Step Tutorial

YOLOv8 is a unified computer vision framework that supports three tasks—object detection, image segmentation, and classification—in a single model, offering faster training and more lightweight architecture than previous versions. To train YOLOv8 on custom datasets, users must prepare data in the YOLO format with specific folder structures (train/validation/test containing images and corresponding label files) and a data.yaml configuration file specifying dataset paths and class names. The model can be trained using either command-line interface or Python scripts, with training commands requiring parameters like task type, model variant, data configuration, epochs, and image size. After training, the best.pt weights are saved in the runs directory, and the model can be validated and used for inference on new images.

Click to watch →

Train Yolov8 Object Detection Python On Custom Dataset Roboflow

Train Yolov8 Object Detection Python On Custom Dataset Roboflow

12.3K

YOLOv8 (You Only Look Once v8) is a single-stage object detection algorithm that identifies and localizes multiple objects in images by dividing them into grids and predicting bounding boxes and class probabilities in a single forward pass. To train YOLOv8 on custom datasets, developers need to: (1) install Ultralytics library (pip install ultralytics), (2) prepare datasets with train/validation/test splits and corresponding label files, (3) configure training parameters including model size (n/s/m/l/x for nano/small/medium/large/extra-large variants), batch size, and epochs, (4) run the training command with appropriate paths, and (5) evaluate results using metrics like mean average precision (mAP), confusion matrices, and loss curves. The trained model can then be used for inference on new images or videos to detect objects with specified confidence thresholds.

Click to watch →

Train Custom Flowers Classification Model using YOLOv8 | Complete Deep Learning Tutorial

Train Custom Flowers Classification Model using YOLOv8 | Complete Deep Learning Tutorial

127

This comprehensive workflow covers training custom YOLOv8 models: (1) Select appropriate model version (N, S, M, L, X) based on parameters and input size; (2) Organize dataset into train/valid/test folders with class-subfolders; (3) Configure training parameters including image_size, batch_size, epochs, and early stopping patience; (4) Execute model.train() with specified configuration; (5) Save best-performing weights using save_period=-1. The process transforms raw image data into a functional classification model through supervised learning.

Click to watch →

How to Train YOLOv8 on a Custom Dataset (Free Google Colab Tutorial)

How to Train YOLOv8 on a Custom Dataset (Free Google Colab Tutorial)

264

This comprehensive tutorial covers the end-to-end process of training a custom YOLOv8 object detection model. The workflow includes: (1) Downloading and preparing your custom dataset from Roboflow with proper YOLO format selection; (2) Configuring Google Colab with GPU resources for efficient training; (3) Setting up API credentials and installing Ultralytics library; (4) Configuring training parameters including task type, dataset paths, base model selection, epochs, and image size; (5) Executing training and monitoring loss and accuracy metrics; (6) Saving the trained model as best.pt; (7) Testing the model on new images to evaluate detection performance. The model's accuracy improves with more training epochs, and results are visualized through detection images showing bounding boxes and class labels.

Click to watch →

Edge AI Intro

Edge vs Cloud

Jetson Lineup

Hardware Deep Dive

Model Optimization

Performance Benchmarks

Technical Q&A

Advanced Usage

Final Resources

Edge AI Intro

0:00

Playing Section

1
Introduces the session on YOLO performance at the edge.
2
Lakshanta presents Seed Studio's focus on embedded AI hardware.
3
The partnership between Ultralytics and Seed Studio is established.

8.2K views14352:54@UltralyticsOriginal Release: 2023-03-29

YOLOv8 can be efficiently deployed on NVIDIA Jetson edge devices using TensorRT optimization, which enables real-time object detection by leveraging hardware acceleration. The Jetson lineup ranges from the Nano (0.5 TOPS) to the AGX Orin (275 TOPS), with performance scaling non-linearly with TOPS. TensorRT export in FP16 precision doubles inference speed with minimal accuracy loss, while INT8 quantization requires calibration to maintain accuracy. The Ultralytics and Seeed Studio partnership provides optimized hardware-software integration, making edge AI deployment accessible for applications requiring data privacy, low latency, and cost efficiency.

[00:00:00]okay all right we are live we are all set okay guys hello everybody uh this is our latest Ultra Linux live session with Glenn myself and like Shanta here like Shanta is an engineer at Seed Studio and he's here to tell us a little bit about uh how yoloviate is performing on edge devices like the super popular Justin Nano so why don't you introduce yourself and tell us a little bit about Seed Studio yeah thank you Glenn I'm very excited to uh attend this uh uh talk today so actually I'm uh I'm lakshanta and I'm an application engineer at Seed Studio so I mainly focus on embedded systems iot and Ai and also in my work I develop different demos and do the documentations tutorials for those products so I have a bachelor's in electronics and information engineering and after I finished my bachelor's I joined Sage studio so mainly I use I I work with Arduino and Raspberry Pi products like I used to work with Arduino and Raspberry Pi products before but now my main focus uh has been the Nvidia Jetson products so so uh yeah so now I would like to introduce a little bit about Seed Studio uh so uh I will just share my screen right now uh yeah so actually seed uh Seed Studio is a I already had their partner and we we started in 2008 uh and we have partnered with more than 470 000 Global iot developers and we have also a closely worked closely with the leading chipset companies uh to provide thousands of Open Source uh us modules for rapid prototyping and also here at scale their projects with agile Manufacturing Services and global distribution networks so actually we recently uh announced this roadmap of an advanced perception system of modules and devices and solutions so uh on the Raspberry Pi uh day the Pi Day the 314 right so that's when we had this uh uh event where we announced all our exciting products that are releasing in uh 2023 and we aim to bring uh system integrators a full toolkit of sensing networking and Edge Computing to leverage the digital transformation for Industries and these releases on uh will be available uh in the coming months in 2023 and at the end of this webinar uh so that will be one of our takeaways so I will uh give like a link to this we have kind of like a product catalog where you can go through all the products that we will be releasing uh in the coming months so uh and also uh we have focused uh on embedded AI now because uh previously uh we have focused a lot around the Arduino and the embedded system like Raspberry Pi's but now we want to focus more on the Nvidia judgment product line because we we think the AI is the future and embedded HD AI is what we want to focus on so we actually uh an Nvidia a life partner and we also provide full product line from you can see Imaging sensors sliders and development kids career boards and full systems so we also address custom Solutions means through uh Hardware customizations and also we partner with is fish and algorithm communities to drive ever growing AI applications so that's what we want to achieve uh in 202 and three and also I want to mention that we have partnered with your ultralytics a couple of months back I think with uh we with yolo V5 so that's right yeah we did some we did some integration on the YOLO V5 with the YOLO E5 on the Jetson Nano and the Xavier NX and we also prepared some you know tutorials on the official ultralytics uh documentation so so I think uh now uh it's time it's it's nice to be back uh to talk you know together with Glenn and your with ultralytics AI uh Solutions yeah absolutely this is uh this is all pretty exciting and interestingly enough my interactions with C Studio personally and ultralytics uh go back much further than YOLO so ultralytics initially got started doing particle physics for the US government we worked under a few different intelligence agencies on anti-neutrino detectors and as part of that we started building uh our own hardware for these detectors which included circuit boards so I got very familiar with a circuit board layout tool called Altium and I started creating these little circuit boards which eventually got more and more complex and uh when I was searching for manufacturers for these pcbs printed circuit boards as they're called uh C popped up and uh they had a few different offerings there and uh and so we started getting some circuit boards back from seed a long time ago I think this was back in 2016. uh so this is in the very early days of ultralytics before we had ever got into AI and I remember I was very surprised when I started Yolo V5 and a few months in somebody asked the question and I I went over they provided a link to a seed forum and I went over and it looks like Steve had like a demo prototype for like YOLO B5 running on edge device so I was really excited uh because uh we sort of reacquainted ourselves in a completely separate business category with completely separate technology uh so so it's really exciting and uh and I'm a big fan of what seed is doing here on on the edge devices so uh the most popular Edge devices are Raspberry Pi's probably uh these are so cheap but they don't come with gpus so you can run YOLO models on these um and you'll get some results but you won't be able to for example run a YOLO model in real time from a streaming video or you could but you'd have to sample very sparsely maybe like one frame per second at a low image resolution and so when videos come along most famously known for uh very expensive uh energy hogging gpus that cost thousands of dollars and plug into your computer to help you train models but they also have a lineup of low power devices uh Jets and Nano is the lowest power and the smallest one of these so it's more expensive than a Raspberry Pi but not much so I think it it comes in at about a hundred dollars uh and you can you can get these from a few different places but seed has done an amazing job of going one step further so seed offers kind of Hardware combinations uh that pair for example Raspberry Pi's with Jeff's and Nanos and they also take care of what's called like the firmware and the software so this makes it much easier to get started with yolo and we've worked closely with them ultralatics and see together to make sure that YOLO V5 models work out of the box when you buy the Jetson Nano from seed in that they're optimized so you get the best performance there's no simple task and I think seeds done a great job in uh in this partnership to to make sure that people get started really quickly with yolo which has always been one of my Big Dreams is to just make this so easy that it just works kind of like what jobs used to say I don't think we're there yet it's sort of uh like a mirage in the distance but we're heading in that direction and it makes me happy so I'm a big fan of what seats doing and uh and I think just being in division States itself really lends the YOLO models to run on edge devices which is really different than what you see for example in language models that are like so big and so complex that they need to be run in the cloud typically through apis but in the vision space companies can run their own models they can run them right there on the edge even in very low power devices so pretty exciting uh people running YOLO models everywhere in the cloud on edge and we're making it easier than ever and uh lexantha's here to actually show some benchmarks I think so these live demos are always like a little risky but I think he's gonna he's gonna show us a little bit about how this works he's going to share your screen and we're going to cross our fingers in and make sure that goes through it yeah so I think before moving that to that I think we can talk a little bit about Aji what do you think okay yeah yeah let's let's do a little more intro then tell people understand yeah so yeah I think some of our viewers are experts but some of them are just getting started too so it always helps to kind of start from the basics and explain uh like how this got started and the benefits of edge AI also overcloud yeah so I I think like the news Aji I think I think one of the most important thing that we want to focus is about the data security and privacy right because uh because everything is happening at the edge and you're not you're not sending any data to the cloud so and also like you can have a low latency with with real-time analytics so that is something that also uh can achieve uh when you run at the edge and also you can reduce the internet bandwidth right because normally like if you wanna because earlier like if you wanna like uh perform AI you want to like you know maybe subscribe to a cloud provider where the inferencing will happen on the cloud so that time you need to provide maybe they have like kind of like a subscription basis plan where you want to like pay them and then you know uh get their service but running at the edge can really like cost like make your cost down as well yeah so so I think these these are one of the like uh advantages that I see uh you know running AI at the edge and uh Nvidia uh Jetson like really helps to to achieve that yeah yeah uh yeah yeah absolutely there's a number of different benefits here privacy concerns are of course like uh one of the main ones you don't have to send your data anywhere you you get the result right where the data exists uh there's also latency like you mentioned there's also cost uh if you have a small experiment it doesn't cost you a lot to send those pictures of the cloud get those images processed and get the results back but when you're considering stay scaling like a very large solution to maybe thousands of cameras uh then your Cloud expenses are are definitely going to rise and we've had uh experience with this ourselves in the early days of of our website and of Yola we put an API out there so people could upload pictures to our website and see results with yolo and we built this like Auto scaling Cloud solution it was really cool it worked too well actually somebody figured out how to send a lot of pictures and uh Google scaled our Cloud instances a lot and we ended up running millions of inferences in a few days uh and we got a bill for thousands of dollars so uh the cloud Solutions are very scalable but they're also uh potentially very expensive and running devices on the edge allows you to control those costs put a cap on them yeah yeah yeah and also like uh yeah I think uh we had another uh webinar before uh a couple of months back Elaine from seat studio uh we she attended this YOLO Vision uh uh yeah that conference that's right yeah you're looking after a big event every September yeah there's the first one last year yeah so the next one is coming up September 2023. hopefully you guys are making another appearance yeah so I think we we can maybe attach a link to that webinar later in uh for at the end of this uh you know uh this YouTube video so that so that oh yeah that's a great idea because alien actually give a lot like more more uh uh insights into all the hardware all the products right so so that would be more helpful for the community uh to learn more about the uh Jetson products that we offer yeah yeah um yeah today so actually I have a product with me our latest uh Jetson product so it is actually based on uh the Jetson Orin NX so so you mentioned yeah this is actually the latest this is here so can you tell us a little bit about the entire Jetson lineup and what the different price points and powers of each device yeah for sure so actually Jetson started with uh the judge and Nano that we that normally people that's what like if someone want to step into the Jetson ecosystem the first Jetson that would they would get their hands on would be the judge and Nano so so that is where it starts and then it went to the Jetson Xavier and next and then came the Jetson TX to platform tx2nx and then after that uh now the latest uh uh platform is suggestion Orient platform right so it is based on the Ampere GPU architecture the Nvidia ampere architecture and uh if you look at the the tops performance right the tops so it would be like the Nano would have 0.5 tops and the Xavier index would have like 21 tops right but the urine uh orange Series has Orin Nano so I can show you like uh Warren also comes in Nano size yeah so uh okay yeah so I'll show you the yeah this one uh okay so this is the orange Nano yeah so actually the orange Series has already Nano uh has two versions 4GB and 8GB uh with 20 tops and 40 tops right and the orange has 8 and 16 JB with 70 tops and 100 tops and the Ajax Over N which is like the biggest uh which is the highest one it says 200 tops the 32 and the 64GB has 275 tops right so so you're saying the original Jetson devices had 0.5 and now the latest the Nano is 0.5 yeah Nano is 0.5 yeah soon video offers the Nano for sale still but they also have the Justin Oren Nano also exactly exactly okay right yeah but the the difference in performance is is very much compared with the the Jetson Nano and the orange Nano because urine is an entire different uh architecture right yeah so the way I usually read this is uh I'm more familiar with iPhone neural engines and so Apple's been doing a great job there also on increasing the tops and I think they're a little under 20 maybe like 16 or 17 these days and that that is enough to run the biggest YOLO V8 models uh at real time like on our IOS app for example so 20 tops is a lot of performance uh and I'm guessing it's something that costs a lot less than an iPhone yeah yeah and so so what I have now is our latest uh recomputer j4012 uh which is based on the Jetson uh Orion NX 16 JB this particular module right so this is actually the the device I have right now so as I said like we provide the entire like uh full systems right so it comes with this beautiful enclosure and uh so once you open it you see the you see the device inside um so it's basically like mounted on this on this board and I have removed all the screws so you can take it out from the from the enclosure and then if you take it out uh we can see uh the device itself right so basically this is the system on module this this this one with the heatsink on top of it and then what we do is we manufacture this career board this this career board this is what we manufacture and to support the the system on module and then we provide the enclosure and we we make it as a full system uh where it it's very easy it's ready to get started with with the product and also uh so you can see there's an SSD uh as well the NVM ss3 so so like when you buy this j4012 it comes with the Nvidia jetpack system already uh pre-installed right so uh hello yeah can you hear me yeah I can hear it yeah okay so when you buy this uh device it comes with the SSD already pre-installed with the jetpack system so because normally like when you buy when you get a Jetson device uh sometimes uh you need to get around yourself uh installing the flashing the system to the device and it takes a lot of time and get time to think around so when you when you have this kind of solution so you just plug it and then SSH into the device and you are ready to go with the device and just start deploying all your applications into uh the device because this will come with all the uh necessary uh like libraries like uh like Coda tensority code enm and all the pytos and everything will be like already pre-installed on this particular uh SD right and uh yeah so so you can see like it has like four USB 3.2 connectors uh one HDMI there's a power connector and there's a ethernet connector and also it has two CSI 15 Lane vpcs camera connectors right because you know if you want to do some visual projects some people want to use maybe a USB camera into the device but there will be some applications where you want to have a compact system where you have some CSI cameras connect to this right and then I have the camera connecting outside so that you will not be using the external uh connectors right so that's why the CSI characters are included to make it a compact maybe like if you want to make a robot with this device right then you can make a compact device including the CSI cameras for the detection purposes and also it has this uh the m.2 key so that you can have Wi-Fi connectivity you can connect Wi-Fi Bluetooth module into this slot so that you can enable Wi-Fi and Bluetooth on this device uh and also it has a 40 pin gpio as we all know it also comes with the Raspberry Pi and all most or like most of the embedded devices come with this kind of 40 pin gpio right if you want to connect like some external sensors uh to interface uh with with this device right so yeah and uh what else yeah it has a can can connect if you want to do some cam projects okay and At Last I think it has an RTC right so you can just connect a battery into the device and uh yeah you can uh yeah that's pretty much it about the about this j4012 the recomputer j4012 which has like 100 tops uh AI performance and also at last uh yeah it also has like uh FCC CCE uh rohs and ukca uh certifications so if you want to like go into production you can easily go like into production with all these uh certifications uh or with already in the device especially for uh for Industries right yeah so yeah okay so it seems like everything is like really tightly integrated there and yeah it's almost plug and play uh what kind of cameras can we connect to this yeah so uh we have tested with the IMX 219 cameras which which is quite popular in the in the embedded world like people used with the embed devices imx219 and also imx477 cameras yeah uh when it comes to CSI cameras right and maybe uh there would be some other cameras uh if you want to use with this device maybe it will not work um it maybe it's because like it will need an additional driver right so if you load the driver into the device then for sure uh that camera will also work but for the IMX 219 it will work out of the box uh after you connect the camera and for the USB ones uh yeah that's fine like you can connect any USB camera uh it will work as long as the USB camera is uh is supported by Linux yeah yeah okay so this is really uh like a computer all in one like you've got a CPU a GPU memory internet connection everything in here exactly so it got 16 GB uh ddr5 LPD data 5 memory uh with the with the GPU the Nvidia uh yeah so that that's the one which can achieve the 100 tops and actually for the storage uh yeah it got the SSC right yeah okay okay you need to install this because uh earlier Nvidia used to have the emmc on the system or module for example the Xavier next in the past uh it had an emmc for the production module but the production model of Orin NX does not have emmc they uh that's why like Hardware partners are like us so this is a really powerful device that you can get uh quite affordably at uh really tremendous performance here at 20 tops yes um you can get started uh right now so seed has this uh available right now right yeah yeah yeah you can visit the yeah I will attach uh a link uh how can I um yeah so if you go to sagestudy.com yeah right yeah with three years yeah and then yeah you can okay so let me ask you a question the models then that are running on these devices we want to export these into tensor RT format right just like any other Nvidia GPU okay exactly actually tensority is very important when it comes to uh uh the uh the Nvidia Json products because like we are talking about uh like uh running AI at the edge and using utilizing tensority on the Nvidia Jetson will make sure that uh all of the hardware Computing is uh used yeah otherwise right right otherwise we will like just uh you know uh so that's why I think tensority will be very useful yeah of course I guess it's just like technicality but we can run really any format uh but if we export the tensority then we'll be optimized uh for these Cuda Edge devices exactly the difference will be like pretty yeah yeah substantial yeah for sure yeah yeah absolutely we've we have YOLO benchmarks on YOLO V5 and YOLO V8 that we run every day uh on different devices different hardware and of course what we see is that for Cuda devices you can't beat that to our team it's going to produce the best results um we natively offer of course fp32 but also half Precision fp16 export and I think we're working on in data exporting so this uh this is this raises an interesting question though tensor RT is the only format in Yolo that requires you to export on that specific Hardware so in this case uh we would need to transfer our YOLO model to the edge device first and then run an export 2103 format locally on the device right to get the the full benefit of the trt export yeah otherwise actually yeah you need to uh also for uh serializing the engine file uh you need to serialize on the on the device that you're gonna deserialize as well for example if you serialize the engine on an X-ray NX device you cannot deserialize the engine on the Orion NX it would like uh if it would not deserialize so the you need to make sure that the engine is generated on the device that you want to uh you know and again do the inference yeah yeah I've always been curious about this uh since I'm not exactly a trt expert if I have one horn device and I I export my uh my YOLO model there I have it in trt format can I then take that trt model and let's say I have a thousand cameras deployed in my company can I just use that and deploy that same model of these thousand devices or uh even the same device yeah same exact device yeah uh okay okay so there's no problem there it's just that no problem as long as the device uh the same right as long as the as I told as I showed you before the already Xavier NX or in Nano uh AGB Arena origin and next all these different uh architectures different uh yeah architectures uh if as long as you you ex you deploy it to the exact same architecture on another device it's going to work for sure yeah yeah okay okay what about the what about the Last Frontier here which is it eight export uh we supported export in a few different formats but it's always a little tricky and some of them require uh more information like for example the images that you're going to be running on to really get Dynamic ranges for quantization is trt into export simple or uh like how how do you describe the process what do you guys usually run uh your models with their red seed are they models fp16 models yeah actually for now we are running the FP fp32 uh just fp32 okay yeah actually yeah we but we have done benchmarks for the FB 16 and 88 as well we have just done the benchmarks right but uh I think for INT 8 what is uh what is important is to do the calibration right that's the step uh for the calibration for the Precision so so I think as long as you do the calibration uh you can you know uh okay run on in eight Precision right yeah okay so the calibration is an important step then okay right and this is because because yeah because otherwise uh you will lose accuracy we've seen this before you know retain the accuracy otherwise uh because as we move down like as we try to you know uh make it fast there'll be a little bit of compromise right because yeah when you're coming from SP 32 to 16 to 8 as we uh you know uh do the make the latency low uh there will be a little bit of compromise in accuracy so to to reduce that we are we do have to do the calibration process yeah yeah that's a good point uh a lot of the times in AI or in Engineering in general it's not so much about filing the very best uh performance but about finding the right compromise so what we see in the AI space is that as we get into quantization uh in pruning and even quantization we're training oftentimes you know it makes sense this sacrifice say like one percent inaccuracy if you can double your speed so I think a lot of people would be happy with that sort of compromise and that's that's sort of what happens once you discretize to and date uh of course within date you've only got 256 different values that a parameter can take and because of that uh to get the right dynamic range safer lookup table uh I think it requires a bit of an extra step that you don't have with that p16 fp16 can still represent a pretty wide dynamic range and what we see in our benchmarks at least on CPU is that fp16 performance is identical uh really to fp32 Performance but yeah going beyond that it's going to end date is a little more tricky and we also see though yeah yeah just going to fp16 alone will double your speed for the most part uh so it's a and there's there's almost no accuracy lock there or no no real negligible kind of uh materially accuracy loss so that's a really easy decision to make uh so I would I would recommend anybody that runs to models anywhere especially on edge devices of course at a lower power it shouldn't be exporting at half precision and we make that really easy with yellow V8 you just use a half argument so you just say YOLO export and just say have uh and if you're running that in Python just say half equals true it's that easy yeah and also like I will quickly show you the device that I mentioned before this is the okay j4012 device that I was talking about the one with the 100 tops and if you go to seedstudio.com uh and search for j4012 you will be able to find the device I just uh explain uh showcase a few minutes ago yeah so this is what about the what about the very entry level or a nano device the four gigabyte one let's see yeah so the numbers I think that one might be more popular yeah so the Nano uh is this one right so we have the Nano as well okay that's a dev kit right that's the most basic one right okay for 149 that's pretty good yeah yeah and yeah so yeah to get more performance yeah you need the payment also yeah okay yeah okay so let's see so this time has been flying here so I think let's see I think he has some benchmarks he'd like to share and for sure for sure that will pick a few of these uh questions from the YouTube channel if you have any questions drop them for us there in the YouTube chat and we'll pick a few to answer yeah so yeah so I have been doing some benchmarks last week uh okay so let me share that okay let's see are these available uh like on a website too uh uh I I'm not at the moment I'm I'm trying to do a Blog so he's on social media yeah I think uh yeah right or maybe Twitter yeah LinkedIn and Twitter just uh but okay but I will make like a Blog maybe yeah this week yeah we should definitely make a vlog yeah I think we usually would love to see this okay so all right we've got some size models X is the biggest on the top and then n is this so this is the Ajax or in uh uh okay I cannot see this uh all right okay so this is 32 GB the device and you can see that these are the models YOLO V8 in I Benchmark from V8 into s to M to L to X right yes yeah five sizes and uh I use the trt exec uh tool that comes with the the tensority when you install tensority on the Nvidia adjuster device this trt exec tool will also be there I think it's not just for the Jets and also for like like the Lena just the basic Ubuntu if you install the tensor if you have in sensority the plpx82 will be included inside right so okay uh using that I have benchmarks so you can see that as we move up uh to YOLO V8 X So within eight Precision uh we can have like a 75 uh tops right yeah which is uh FPS yeah for instance which is pretty impressive for foreign YOLO models included in the Alternatives right so I have directed chain models yeah exactly yeah I have downloaded them from the website the ultralatics and then I've benchmarked them uh over here so I think as Glenn mentioned before there's not much of difference between yeah so uh you can see the pie torch so first I I did the pie torch test without any tensority optimization you can see if you compare this with intake there's a lot of difference yeah there's a drastic difference the date is really important and actually like what we see here at yellow 8X is going from fp32 to fp16 is exactly doubling the speed like like what we've also seen so you double your speed by going to half precision and uh you can maybe improve it by two-thirds not quite double it but uh but do even better at indeed so so it seems like the the clear solution here is CRT exported in date with the exactly the appropriate calibrations so that you don't lose too much accuracy in that in-day transition yeah and also this is what uh yeah this is uh what you can find inside the alternative repository right so these are the images these are the models that I downloaded and yeah as you can see like yeah so and these are trt profiles also right here in the speed a100 column uh yeah these are very fast uh but they're also running on an a100 GPU so these are also at 640 image size batch size one right right and uh oh yeah and I have this one as well so I uh I checked on three devices right so this is the orange 32GB and the Orion nx16 JB so this is the one that I showed uh a few minutes ago the recomputer j4012 with uh 100 tops and this is the JF j2021 with 21 tops so we have 21 tops 100 tops and 200 tops right so so that's the so we can have a comparison between the devices uh for the Benchmark it's interesting there's not uh there's not a linear ramp in speed with a linear ramp and tops it looks like there's a maybe like a floor some operations that uh that aren't like fully optimized to I guess exploit additional tops yeah yeah because uh if you see like uh yeah you are right that's not linear uh yeah I think I think that makes sense though uh it's not quite that easy like for example the the very small models like the Nano models also like you'll start to see or what I've noticed or what I've observed is that uh if we have a model let's say half the flops it doesn't run half as fast so it's not uh in general those two things correlate together and it looks like the tops also uh correlate but uh not sometimes as you'd expect but uh but we definitely see improvements here as we increase the tops and for the higher power devices we definitely get more frames per second for sure for sure and also I have this another type of graph because for this one uh like we are running different models on a single device and doing the uh graph right so I have not plotted another graph where you have one model and you compare three devices right um yeah so it's a different view of the same data then different beef yeah for sure yeah so we have the Xavier NX 21 tops 100 tops 200 tops three devices really informative each uh each model right so so we can realize it this is really good stuff we definitely have to write a blog post and I think share this on the repository this is very informative I think it'll answer a lot of questions yeah I think for sure we can do you can we can officially post on the uh Ultra analytics documentation right yeah oh that's the other thing yeah benchmarks yeah yeah that's something I'd like to say we got a lot of criticism in yellow V5 for having weak documentation and uh the criticism was correct so we've done a lot of work now with yellow V8 we've improved the documentation uh substantially we put a lot of effort into it so it's it's much more informative it's a much better resource and we're continuing to improve it too so I think this would be great to add to the docs also right um yeah so this is the biggest model the YOLO V8 X model comparing with the three devices right so okay yeah so 24 46 and 75 with 21 tops 100 tops and 200 tops yeah four and eight yeah okay yeah yeah so and also like one more thing uh there's this uh I showed you before the agx this particular device the agx orange developer kit right so that is a that is also by that is actually officially by Nvidia right uh that developed it and if if you have the developer kit you can emulate all these uh uh devices so you can emulate a nano uh four eight eight sixteen all these six things can be emulated on that device so so so what is the benefit so if you want if you want to uh maybe uh test a particular module uh model uh on the device but you know you still don't have the device but you have a agx orange dev kit so you can verify the device and you can make sure what exact device that you want to employ for the for your uh like industrial or other like commercial application right so maybe okay uh if you have a orange index 8GB so many tops maybe that's too much for your application so maybe then you're going to emulate or in Nano 4GB 20 tops I think so yes if you don't know which Hardware you want you can exactly or in developer kit you can emulate the different ones and uh see the the minimum that you require uh for your application and then go ahead and purchase those save yourself some money okay very interesting it's very useful for the industries actually if they want to maybe deploy like 1 000 like like you know Jetsons uh deploy at scale then this will be a really good starting point uh you know to do the emulation at first yeah okay very interesting yeah okay like stop it thank you for these numbers it's been uh really interesting actually I've learned a lot myself learning every day actually so I didn't actually know any AI until about four years ago that's when I started getting interested so I'm pretty much self-taught and all this and it's by seeing demos like this and uh just learning from the experts okay so I'm going over to YouTube and I see we've got a few questions here so uh all right let me scroll down here and let's see okay so uh opaluso is asked about quantization on edge devices and should we expect the yellow V8 papers soon hopefully including some benchmarks yeah so the paper is a Hot Topic I think I get asked this every live session the reason there's no paper there's two reasons so the first one is that uh we're focusing on adding features and fixing bugs in Yolo uh so we are a small team and we're trying to do a lot and I think the most value for the users is by adding new features so we've been really working on a lot of these uh most recently tracking now pose estimation and uh also if we had written paper for example at launch then we'd have to update it uh so since things keep changing keep improving and we keep adding features uh we're waiting until things settle a bit to really focus on the paper but for quantization uh yes we have a export tutorial for YOLO V5 that covers quantization and we're going to be migrating that and updating that for trt and also into quantization in a number of different formats including ones like TF light and core ml okay uh let's see here do we have okay let's see Dragon Sage says does yellow V8 support nms natively on these devices that's a good question uh so for most tasks except classification you're going to need ATMs which is a non-maximal suppression module so different export formats handle this differently for some of them uh we can pipeline these together into one model like core ml does this really well and we have an nms argument during yoloviate export that you can pass and it'll automatically pipeline that for you other formats don't support pipelining yet and for these formats we apply Forge nms so you can for example export a tensor T model and once you have that you can load that up again with YOLO you simply point to the trt model instead of a pi torch model it'll load up just like a pi torch model and you can run inference with it it'll be accelerated but it's still using the torch nms which will be GPU accelerated so uh we're working on I think adding Onyx support for that and possibly tensor RT support for that also this would allow you for example to deploy a solution without needing the torch dependency which is uh pretty heavy so for very small Edge devices there's added benefits if we can work on that if you have expertise on that please contribute we'd love PRS okay let's see uh let me see if I can find a question that's the question yeah for me yeah oh yeah I agree he's asking yeah other new Jetson devices support power from 4D pin right so yeah so that's a good question actually uh uh the problem uh is uh the the new Jetson devices need at least a 12 volt at 5 amps right it's so for example the Orion NX device that I showed you before based on the ray computer j4012 that device needs 12 volt at 5 amps so the problem is uh the the power uh the maximum current from the pin can uh cannot have 5 amps which means you cannot power in from the uh from the 4D ping so so so it's not possible yeah you need to have you need to correct the the barrel Jack uh to the board to power the device that's the only way yeah interesting okay so okay so I had a question about that actually I was thinking of the the battery connection that you mentioned yeah so that's the RTC right that's the RTC battery okay and yeah uh so you need a 12 volt battery uh like a Lithium Polymer battery you could hook up to that and then you can I don't know you could run this remotely the battery is not for the RTC like that that's just to retain the time just like uh like uh oh it's a very small battery oh I see okay yeah the coin cell one right that's just to return the time and the date right okay let's think of a different type of battery that would run the entire thing I don't know it's the same thing that you have in on the motherboard but apart from that if you want to power the device the only way to power it is through the the barrel Jack connector uh on the okay on the board which needs some power uh through 12 volts at five amperes yeah so because it's running at like 25 it can have 25 watts maximum power so okay yeah 25 to 30 yeah so yeah yeah and let's see here okay I'm gonna pull one last question here uh oh actually let's see Brian Merritt has a good question he says are there examples of transfer learning for YOLO V8 I would like to save GPU training Time by adding a new image uh or set of images specific to individual applications to existing models that do uh 90 percent so okay the term transfer learning is uh it's a bit of a blanket term uh typically it's meant to imply that you start permit retrained model rather than starting from scratch uh and in that sense this is the default use case for YOLO V8 so the tutorials that we have at the docs and everything you'll see it'll always show you loading up say an official model and then training that model on a new data set uh so in this sense what happens is if you have say a data set of cats and dogs we'll strip off the output layer of the pre-trained model since it'll have different number of classes and we'll create a new head that's randomly initialized so your model will be trained uh the majority of the model will have really learned low level features things like edges colors textures and this dramatically reduces training time uh and of course this is all happening automatically um and it's the default use case really you can always start from a scratch model but we wouldn't recommend that sometimes the question is uh maybe like one passes and the question is uh I like Coco data set I just want to add a class to that like I wanted to also learn choose and the answer to that is that there's something called catastrophic forgetting so the entire model is applied to learning your data every parameter in the model is focused uh we get gradients from them and we update every parameter so that it's 100 applied to improving on your specific data set that you're presenting to the model this means that as it learns the new data if you don't keep presenting it the old data it's going to forget that data and that's actually a benefit because that means that every resource inside the model can be dedicated to the data that you're presenting to it uh otherwise you'd have subpart training and so the short answer here is that you can't simply add a classical model uh you can you just have to combine both data sets that's very easy YOLO you just put them both together you have to make sure that the classes don't overlap so you want these classes to be complementary so for example if you want to train on Coco in another class you simply label your new data set with classes starting from index 80. since Coco has 80 classes um so it's a little complicated but uh that's the basic answer transfer Learning Happens automatically by default uh and if you'd like to add class into an existing data set train on both making sure that the classes don't conflict okay and one last question here that's a pretty good one by fifth person he says how can we deploy yellow V8 on Android devices running on the edge okay this is a great question so uh very soon here we are making YOLO V8 available on HUB Hub is our no code deployment tool just at hub.ulturalytics.com this allows for automatic deployment to Android so you can train your model you can see it on our demo app and I know fifth person's next question is going to be that's great but how can I use this on my app and we have an answer for that we are creating Frameworks that will be open source and you can just copy that code and the model will work directly in your app so we're going to make it real easy for you to also use the trained models directly in your custom app uh on Android and on iOS also with TF Lite formatting core ml so that's it and uh wow this one really uh I'm talking a lot okay we're almost up to 50 minutes here it's supposed to be 30 minutes but looks like we had a lot of fun stuff to talk about all right like Santa yeah and also I think yeah I think I would like to share one of the tutorials that we made last week absolutely so just I think to close off our discussion I think yeah I can yeah show off the tutorial so that others can also uh you know access that so this is the tutorial that I did last week to deploy YOLO V8 on the Nvidia Jetson using tensority and deep stream SDK right so you can visit this link uh so this link we can attach in the YouTube description other others everyone can follow this link yeah so you can follow through uh the tutorial and also it has it has like the the steps on how to use the trt X sector as well so that uh everyone can you know benchmark uh the models by themselves as well so okay well this looks really informative yeah so this is actually based on uh yeah so based on the Deep stream uh SDK so and also uh finally uh so as I mentioned uh I will uh so I have this uh this slide sorry okay slide anyway if you if you paste me that link in our chat I think we can paste it in the YouTube chat and uh let everybody else see this too oh for sure uh the the wiki right yeah this one yeah this wikileak it looks really interesting I see we have more questions too I just scroll to the bottom here yeah I will attach the VK link uh yeah so no this far oh okay I've got a really good question I can answer here so Joy Timmerman's is asking about batch sizes so we've got uh two really good options here so tensor RT by default will export at a fixed size this means that uh you have to pick your image size and your batch size at export but we've got this really cool argument it's Dynamic so if you export say YOLO export dynamic uh or in Python Dynamic equals true then you can pass a batch size uh and this will be considered a maximum badge size so if you pass say batch equals 16 then you can now run this exported trt model at any batch size up to 16 and the same with the image sizes it'll automatically accept different image sizes so Dynamic is really cool of course there's no free lunch and I think there's a bit of a performance hit I haven't Quantified it but I'm assuming that it's very small so um so yeah that's your answer uh if you know you're gonna have five cameras for example instead of one uh you should probably export a fixed batch size of five which is also an option you don't need to use batch size one that's just the default during a fixed export uh but if you're not sure how many cameras you're going to use Dynamic argument is the way to go and I think that's shown again in the ultralytic stocks just at docs.ulturalix.com slash modes slash export okay and let's see here uh did you paste that link in the chat for us yeah I did yeah I did basically on the chat uh okay let me see you okay I'm not seeing that did you so you pasted energy in chat oh oh on the zoom okay oh sorry even better directly on YouTube okay perfect all right that's uh yeah I don't see it but if you said you did it I'm sure you did so okay don't you see on YouTube I sent on YouTube HDI the last comment I see is by Dragon Siege uh he just says thanks great discussion I will send you oh yeah and of course we'll update the description also with that so okay super interesting chat uh thanks a lot for joining us like Santa uh from shenza in China over the seed uh headquarters is right yeah for sure yeah yeah thanks a lot Glenn it was a really nice discussion and I think uh yeah so if uh so this is the last slide that I have and if okay you go to this uh this link 2023.c.cc uh you can explore all the devices uh that we are planning to launch in 2023 as I mentioned before so we have a product catalog and you can learn all about the exciting stuff uh that's amazing come out uh soon yeah okay and I'm very happy to be a part of this today's discussion and I think it's it was really informative as well yeah it was uh yeah I've learned a lot personally I'm sure the users have as well all our viewers okay uh thanks a lot like Santa and to everybody listening thank you all so if you have any remaining questions uh feel free to put them here on YouTube or drop them in our GitHub uh as an issue or a discussion so thank you everybody thank you thank you thank you but

Up Next

Calculating TAM SAM SOM: A Founders Framework

@edkang99

960 views•2023-11-10

IFS Therapy Demonstration: Complete Session with Unburdening

@IFSCA

95.9K views•2021-01-13

FastAPI vs Flask vs Django: Choosing the Right Python Web Framework

@TechWithTim

302.5K views•2024-05-26

Game of Thrones Opening Credits: A Cinematic Analysis

@gameofthrones

46.3M views•2011-04-18

Related Study Plans & Knowledge Roadmaps

Structured learning paths in General & Interdisciplinary Studies

Browse General & Interdisciplinary Study Plans

general-studies

Containerization: Evolution, Logistics & Trade

01Ancient Trade & Bulk Cargo: Amphorae to Barrels

02The Industrial Revolution & Cardboard Packaging

03The Containerization Revolution: Malcom McLean

+ 2 more learning modules

Explore Roadmap

general-studies

Textile Tech: Evolution, Labor & Global Impact

01Pre-Industrial Textiles: Handloom & Cottage Industry

02Mechanization: The Spinning Jenny to the Power Loom

03The Factory System & Global Labor Transformation

+ 2 more learning modules

Explore Roadmap

general-studies

Pharmacology: Botany, Patents & Biopiracy

01Foundations of Ethnobotany & Traditional Medicine

02The Birth of Synthetic Chemistry: Aspirin & Penicillin

03The Battle Against Malaria: Quinine to Artemisinin

+ 3 more learning modules

Explore Roadmap

general-studies

Writing Surfaces: Papyrus, Parchment & Paper

01Ancient Foundations: Clay, Stone, and Egyptian Papyrus

02Parchment, Vellum, and the Rise of the Codex

03The Chinese Rag Paper Revolution and the Silk Road

+ 3 more learning modules

Explore Roadmap