Academic Databases
Explore Our Semantic Databases
STEM & Tech
Math, Physics, CS & Engineering...
Economics
Macro, Micro, Econometrics...
Business
Management, Marketing, Finance...
Health & Life Sciences
Biology, Medicine, Neuroscience...
Humanities & Social Sciences
Psychology, Law, Philosophy, History...
General Studies
General & interdisciplinary studies...
Explore Topics
Features

ML System Design Mock Interview: YouTube Feed Recommendations

Added: 2026-08-14

Deep Dive

Basic Machine Learning Foundations: Familiarity with supervised learning, feature engineering, and standard evaluation metrics like ROC-AUC, precision, and recall.

Foundations for Machine Learning | Linear Algebra, Probability, Calculus, Optimization [Lecture 1]

Foundations for Machine Learning | Linear Algebra, Probability, Calculus, Optimization [Lecture 1]

64.3K

Machine learning foundations are organized into five essential pillars: (1) Mathematics foundations covering linear algebra, probability/statistics, and calculus; (2) Programming fundamentals including Python, NumPy, PyTorch, and other ML libraries; (3) Optimization techniques such as gradient descent and backpropagation; (4) AI landscape overview covering historical developments from 2012 to present including CNNs, GANs, VAEs, Transformers, and large language models; (5) Storytelling and visualization skills for communicating results effectively. These pillars provide a comprehensive framework for becoming a serious machine learning engineer capable of solving real-world problems rather than just executing pre-existing code templates.

Click to watch →

Machine Learning Foundations: Welcome to the Journey

Machine Learning Foundations: Welcome to the Journey

285.9K

The Machine Learning Foundation series is organized into eight interconnected subjects divided into four couplets: (1) Linear Algebra - covering fundamental vector and matrix operations essential for ML computations; (2) Calculus - including limits, derivatives, and integrals for optimization; (3) Probability and Information Theory - providing the mathematical framework for uncertainty and data representation; (4) Statistics, Algorithms, Data Structures, and Optimization - covering statistical methods, algorithm design principles, efficient data handling, and mathematical optimization techniques. These subjects form the theoretical backbone of contemporary machine learning approaches including deep learning and AI.

Click to watch →

Your First Step into ML | Learn The Basics of Machine Learning | Great Learning

Your First Step into ML | Learn The Basics of Machine Learning | Great Learning

13K

This section explains the theoretical foundations of machine learning: (1) Machines are good at working with numbers, performing mathematical computations quickly, and remembering past data, (2) Machines learn from data represented as past events and outcomes, (3) Two main learning types: predicting future outcomes from past patterns and detecting patterns within datasets, (4) Three algorithms: Regression Analysis (finding mathematical functions for predictions), Decision Trees (creating flowchart-like decision structures), and Cluster Analysis (segmenting data into groups), (5) Model evaluation using training-test cycles (70% train, 30% test) and metrics like accuracy, precision, and recall, (6) Classification into Supervised Learning (with human guidance) and Unsupervised Learning (independent pattern discovery).

Click to watch →

Announcing the Machine Learning Foundations Tutorial Series

Announcing the Machine Learning Foundations Tutorial Series

2.1K

The Machine Learning Foundations series provides a comprehensive overview of foundational subjects underlying contemporary machine learning approaches, including deep learning and other AI techniques. The eight topics are organized into four couplets: (1) Linear Algebra and Matrix Operations, (2) Calculus 1 (Limits and Derivatives) and Calculus 2 (Partial Derivatives and Integrals), (3) Statistics, Probability, and Information Theory, and (4) Computer Science Algorithms, Data Structures, and Optimization. Later topics build upon earlier content, so the recommended approach is to progress through the topics in the order provided.

Click to watch →

Machine Learning Foundations: Introduction to ML | What is Machine Learning? | ML Basics Explained

Machine Learning Foundations: Introduction to ML | What is Machine Learning? | ML Basics Explained

121

Machine learning requires three essential mathematical foundations: linear algebra, optimization, and statistics. Linear algebra is the core of the course, covering transformations, decomposition, and algorithms like SVD and PCA. Optimization enables building effective models by finding optimal solutions. Statistics allows data modeling through probability, hypothesis testing, and understanding distributions like normal distribution. The course uses an exploratory learning approach where students write Python scripts and discover connections between concepts rather than memorizing formulas. This connected learning approach helps students understand that all topics are systematic and interrelated, preparing them to become top researchers and machine learning engineers.

Click to watch →

Deep Dive

Recommender System Fundamentals: Understanding the core concepts of collaborative filtering, content-based filtering, and matrix factorization.

Trustworthy Recommender Systems | Elena Štefancová

Trustworthy Recommender Systems | Elena Štefancová

105

Recommender systems are personalization tools that output ranked lists of items based on user interactions and item information. They serve to increase sales and customer satisfaction. Data sources include transactions and explicit preferences. Key types include collaborative filtering (using user/item similarity to predict preferences), content-based filtering (using item features), demographic systems, knowledge-based systems (using LLMs), and hybrid approaches. The collaborative filtering principle states that users with similar past interactions are likely to share future preferences.

Click to watch →

Recommended systems unit 1 collabrative filtering

Recommended systems unit 1 collabrative filtering

138

Recommendation systems are powerful engines that power our digital world by analyzing user data and predicting preferences. They appear behind every movie recommendation, music suggestion, and online product recommendation. Unlike search engines that are reactive (only showing what users search for), recommendation systems are proactive—they anticipate preferences and suggest content before users ask. The core purpose is to personalize digital experiences. Four main methods exist: Collaborative Filtering (users with similar tastes will like similar items), Content-Based Filtering (recommends items with similar characteristics to what users liked), Knowledge-Based Systems (for high-value items like cars and houses), and Hybrid Methods (combining multiple approaches for best results).

Click to watch →

🔴 LIVE 🔴 Learning to Rank is good for your ML Career | Learn RecSys with me #4

🔴 LIVE 🔴 Learning to Rank is good for your ML Career | Learn RecSys with me #4

3.7K

Recommender systems optimize utility functions to suggest items to users, using user features, item features, and interaction data. Industrial implementations require scalable architectures with feature processing pipelines, distributed storage (HBase/Cassandra), real-time indexing (Kafka/Samza), and ranking services. Key challenges include the explore-exploit dilemma, data quality monitoring, and balancing competing objectives like engagement and revenue. Product focus and brand consistency are crucial—content must align with platform identity even if engaging.

Click to watch →

How to Design and Build a Recommendation System Pipeline in Python (Jill Cates)

How to Design and Build a Recommendation System Pipeline in Python (Jill Cates)

92.9K

Recommender systems personalize content across platforms like Spotify, Netflix, Amazon, dating sites, job searches, news, and medicine. They emerged with e-commerce, solving the problem of overwhelming choice by filtering relevant items from unlimited inventory. Two main approaches exist: collaborative filtering (finding patterns in user-item interactions without knowing identities) and content-based filtering (matching user and item characteristics). The complete pipeline includes pre-processing (transforming raw data into matrices), hyperparameter tuning, model training, post-processing, and evaluation. Pre-processing transforms raw ratings into user-by-item matrices, calculates sparsity (if above 0.5%, collaborative filtering struggles), and applies mean normalization to correct for user and item bias (optimists/pessimists).

Click to watch →

TSAAI M5-14. Recommendation systems

TSAAI M5-14. Recommendation systems

204

Recommender systems are algorithms that predict user preferences and suggest items based on past behavior. They are widely used in e-commerce, streaming, social media, and other digital platforms. The two primary approaches are collaborative filtering (predicting preferences based on similar users' choices) and content-based filtering (matching item features to user profiles). User-based collaborative filtering finds similar users and recommends items they liked, while item-based measures similarity between items. Content-based filtering uses item descriptors like actors, directors, or genres. Most systems combine both approaches. Other methods include knowledge-based recommendations, association rule mining, and context-aware suggestions.

Click to watch →

Deep Dive

System Design Basics: Knowledge of basic distributed systems concepts, including databases, caching, APIs, and the trade-offs between latency and throughput.

Introduction to System Design | System Design Tutorials | Part 1 | 2020

Introduction to System Design | System Design Tutorials | Part 1 | 2020

577.3K

System design is the process of understanding user requirements and selecting appropriate components, modules, and technologies to build systems that serve specific user needs; systems consist of interconnected components that work together to fulfill particular purposes, and effective system design requires analyzing trade-offs, constraints, and potential failure points to create scalable solutions.

Click to watch →

System Design BASICS: Horizontal vs. Vertical Scaling

System Design BASICS: Horizontal vs. Vertical Scaling

1.9M

System design involves choosing between vertical scaling (upgrading individual machines for faster processing) and horizontal scaling (adding more machines to distribute load), each with trade-offs in terms of single point of failure, communication speed, data consistency, and hardware limitations; real-world systems typically use a hybrid approach combining both strategies.

Click to watch →

LIVE - Aprendendo System Design - Dia 1

LIVE - Aprendendo System Design - Dia 1

1.9K

System design is fundamentally solution architecture - planning how all system parts interact to achieve optimal results. It involves considering functional and non-functional requirements, making technology choices based on trade-offs, and thinking about resilience, scalability, performance, and cost. System design is the engineering of decisions about uncertainties, requiring understanding trade-offs and risks. It rests on three pillars: (1) Decisions - understanding trade-offs without certainty about outcomes; (2) Components - selecting protocols, storage, caches, queues; (3) Constraints - considering budget, deadlines, compliance, and vendor requirements. Unlike merely drawing boxes in diagramming tools, true system design requires deep understanding of why each component exists and how it contributes to the overall solution.

Click to watch →

The Exact 6-Month Roadmap to Get a Software Engineering Job in 2026

The Exact 6-Month Roadmap to Get a Software Engineering Job in 2026

37.9K

Even entry-level software engineers should understand fundamental system design concepts including how requests work, latency, scalability, and reliability. Rather than trying to master everything in depth, freshers should aim for high-level understanding. Practical exercises like designing systems such as Google Docs or games help solidify this knowledge. Working through general system design problems helps develop the ability to think about architecture at scale.

Click to watch →

I spent 1000+ hours on System Design, so you don't have to.

I spent 1000+ hours on System Design, so you don't have to.

117.8K

The first two weeks of system design preparation should focus on reviewing computer science fundamentals. This includes data structures, algorithms, operating systems, and computer architecture. Additionally, understanding the complete lifecycle of a web request is essential - how requests travel from the client through web servers to application servers and back. This foundational knowledge provides the building blocks for understanding more complex system design concepts.

Click to watch →

Deep Dive

Two-Stage Recommendation Architecture: Conceptual understanding of dividing the recommendation process into candidate generation (retrieval) and ranking phases.

Как подступиться к задаче про рекомендательную систему на Avito ML Cup

Как подступиться к задаче про рекомендательную систему на Avito ML Cup

897

Recommendation systems require a two-stage architecture: first, a candidate selection stage that retrieves a larger pool of relevant items (e.g., 100-200 items) using collaborative filtering methods like ALS or baseline models, followed by a ranking stage that uses gradient boosting models (such as XGBoost, LightGBM, or CatBoost) to precisely order these candidates based on user behavior features including click statistics, conversion rates, and temporal patterns. This approach handles large-scale recommendation data (billions of rows) by reducing computational complexity while maintaining high recall and precision.

Click to watch →

Introduction to Ranking and Recommendations | Recommender Systems Lectures

Introduction to Ranking and Recommendations | Recommender Systems Lectures

2K

When recommending from millions of items, a two-stage approach solves scalability: (1) Candidate Generation - lightweight models filter millions to hundreds/thousands using user features and basic item features, (2) Ranking - sophisticated models rank these candidates into final recommendations (20-30 items). This separates computational complexity: generation handles volume, ranking handles precision. The architecture scales efficiently while maintaining quality, with stages evolving from simple filtering to complex neural networks.

Click to watch →

RecSys 2016: Paper Session 6 - Deep Neural Networks for YouTube Recommendations

RecSys 2016: Paper Session 6 - Deep Neural Networks for YouTube Recommendations

18.9K

YouTube employs a two-stage recommendation architecture to handle massive-scale video recommendations. The candidate generation stage narrows millions of videos to hundreds using high-recall strategies, creating a pool where 5-10 videos will be excellent recommendations. The ranking stage then ranks this pool, incorporating additional candidate sources for recently uploaded content. This separation is essential because ranking requires access to video features and outputs the final dozen videos rendered on the homepage. Each page reload triggers multiple inference passes through several networks. The system faces enormous challenges including massive training data volume, strict serving latency requirements, non-stationary video distributions (with videos becoming viral unexpectedly), reliance on implicit feedback due to data volume constraints, and limited metadata from user-generated content.

Click to watch →

Machine Learning System Design - Netflix Recommendation System

Machine Learning System Design - Netflix Recommendation System

10.7K

Multi-stage architecture divides the recommendation process into stages for scalability and modularity. Two-stage approach: (1) Candidate Generation Model - finds top 100 candidates from 100K+ items, focusing on high recall to ensure no relevant items are missed; (2) Ranking Model - selects top 20 recommendations from the 100 candidates, focusing on high precision to ensure relevance. Benefits: Scalability (ranking handles fewer items), modularity (separate models for different tasks), flexibility (easier to update individual components).

Click to watch →

Mastering Multilingual Recommender Systems | Grandmaster Series E9

Mastering Multilingual Recommender Systems | Grandmaster Series E9

2K

A two-stage recommender system consists of Candidate Generation (Stage 1) and Ranking (Stage 2). Stage 1 retrieves item candidates from all items using user history through various methods including co-visitation matrices, vertex embedding similarity, text similarity scores, and neural net approaches. Stage 2 assembles these candidates and performs feature engineering using session features (session length, item count) and candidate features (brand, color, size). This pipeline structure enables efficient processing by first narrowing down the large candidate pool before applying more computationally expensive ranking models.

Click to watch →

Problem Setup

Architecture

Label Design

Feature Eng

Model Design

Evaluation

Deployment

Problem Setup

0:00

Playing Section

1
Design a YouTube recommendation feed returning hundreds of ranked videos.
2
Must balance scale, latency, and accuracy using a multi-stage approach.
3
Define engagement labels and assume offline ingestion for new content.

Deep Dive

Deep Learning for Recommender Systems: Exploring advanced neural network architectures such as Two-Tower models, Deep & Cross Networks (DCN), and Transformer-based sequence models.

Deep Learning for Recommender Systems (Nick Pentreath)

Deep Learning for Recommender Systems (Nick Pentreath)

32.5K

Deep learning techniques are increasingly being applied to recommender systems to improve recommendation quality by automatically extracting features from user and item metadata, with recent advances including deep content-based models, Wide & Deep architectures combining linear and deep models, DeepFM for automatic feature interaction discovery, and session-based RNN models for contextual recommendations, though challenges remain in handling sparse data, computational scalability, and aligning ML metrics with business outcomes.

Click to watch →

Neural Networks for Recommender Systems (PyData 2017)

Neural Networks for Recommender Systems (PyData 2017)

Deep learning offers three key advantages for recommendations: speed of iteration (quickly experiment with model changes without implementing SGD manually), flexibility (incorporate side information like item metadata or images via CNN features), and powerful representation techniques. Unlike fixed-form collaborative filtering packages, neural networks allow flexible loss functions and can leverage visual features from product images. The talk demonstrates transitioning from classic matrix factorization to neural network implementations, enabling rapid experimentation with different ranking losses and sequence-based representations.

Click to watch →

Deep Learning for Recommender Systems | Alexandros Karatzoglou

Deep Learning for Recommender Systems | Alexandros Karatzoglou

10.3K

Deep learning has revolutionized recommender systems by enabling automated feature extraction, handling heterogeneous data types (images, text, audio), and modeling sequential user behavior through techniques like item embeddings (extending Word2Vec to user-item interactions), autoencoders for collaborative filtering, content-based feature extraction using CNNs/RNNs, and session-based recommendations with recurrent neural networks, achieving approximately 30% improvement over traditional methods.

Click to watch →

47th #ebaytechtalk: Deep Learning for Recommender Systems

47th #ebaytechtalk: Deep Learning for Recommender Systems

16K

Information grows exponentially while human decision-making capability remains constant, creating a fundamental gap. The paradox of choice reveals that more options don't always lead to better decisions—research found groups with six jam varieties had ten times fewer purchases than those with two. Clay Shirky reframed this as 'filter failure' rather than information overload, suggesting improved algorithms can help us benefit from available information. Deep learning emerges as a solution because it addresses learning problems arising from this filter failure. Recommender systems operate on three core components: users, items, and their interactions. Collaborative filtering identifies user similarities to recommend items others liked, but suffers from sparsity and cold start problems. Matrix factorization addresses sparsity by decomposing interaction matrices into latent representations, enabling predictions for missing user-item pairs. Content-based filtering augments users and items with explicit features to overcome cold start. Major platforms like Spotify, YouTube, and Google have successfully deployed deep learning: Spotify used CNNs to generate song fingerprints from Mel spectrograms; YouTube employs a two-step pipeline with approximate nearest neighbor search and ranking; Google's Wide & Deep architecture combines linear models for memorization with deep neural networks for generalization. These implementations demonstrate how different architectures solve specific recommendation challenges in production environments.

Click to watch →

Deep Learning for Personalized Search and Recommender Systems part 1

Deep Learning for Personalized Search and Recommender Systems part 1

21.7K

Deep learning transforms personalized recommender systems by enabling distributed representation learning of user and item attributes in low-dimensional dense vector spaces, allowing sophisticated feature extraction through architectures like CNNs and RNNs/LSTMs, combined with gradient-based optimization techniques to improve recommendation accuracy and relevance at scale.

Click to watch →

Deep Dive

Real-Time Data Pipelines: Learning how to implement streaming data technologies like Apache Kafka and Flink for real-time feature extraction and low-latency feature stores.

Building Realtime Data Pipelines with Kafka Connect and Spark Streaming

Building Realtime Data Pipelines with Kafka Connect and Spark Streaming

50.8K

Traditional ETL pipelines store operational data in relational databases and periodically copy it to data warehouses for offline analysis. While simple, this approach has significant limitations: long analysis times that limit responsiveness. Over the past 10-20 years, organizations have evolved toward diverse data systems including primary data stores (NoSQL systems) and secondary indexes (Elasticsearch, graph databases). This evolution requires data to be moved continuously at low latency between systems. The shift toward real-time has driven the rise of stream processing, which handles continuous streams of events rather than batch processing. The fundamental challenge is that as organizations add more systems, they create complex data pipelines where each connection is often handled ad hoc, requiring custom tools and creating significant coordination challenges across teams.

Click to watch →

Data Pipelines en Tiempo Real con Dataflow: Diseño, Optimización y Casos Reales

Data Pipelines en Tiempo Real con Dataflow: Diseño, Optimización y Casos Reales

141

Google Cloud Dataflow is a managed service for building streaming and batch data pipelines using Apache Beam, which automatically handles infrastructure scaling and maintenance while supporting multiple programming languages like Python and Java; it enables efficient real-time data processing through window operations (fixed, sliding, and session windows) that aggregate data over time periods, allowing organizations to process data streams from sources like Pub/Sub and deliver results to destinations like BigQuery with minimal infrastructure management overhead.

Click to watch →

Real-Time Event-Driven Data Pipelines in Apache Airflow

Real-Time Event-Driven Data Pipelines in Apache Airflow

10.9K

Event-driven scheduling in Apache Airflow (introduced in version 3.0) allows data pipelines to trigger automatically when messages arrive in a message queue like AWS SQS, using the combination of a MessageQueueTrigger that monitors the queue and an AssetWatcher that watches for trigger events to activate the DAG; this differs from traditional time-based or data-aware scheduling by responding to external events rather than fixed schedules or internal data dependencies.

Click to watch →

Building a Real-time Data Pipeline From Twitter Into Databricks Delta Lake.

Building a Real-time Data Pipeline From Twitter Into Databricks Delta Lake.

8.5K

This video demonstrates how to build a real-time data pipeline connecting Twitter's fire hose API to Databricks Delta Lake using Decodable as the transit mechanism. The pipeline includes a cleansing stage that filters out tweets without location data (specifically country codes), reducing data volume by approximately 100:1. The processed data is then written to Delta Lake and analyzed in real-time using Databricks, with a sliding window analysis showing tweet activity by country over time.

Click to watch →

Kafka·Spark·Airflow 데이터 파이프라인 실전 구축부터 서빙까지ㅣ+Paimon·Iceberg

Kafka·Spark·Airflow 데이터 파이프라인 실전 구축부터 서빙까지ㅣ+Paimon·Iceberg

175

A comprehensive real-time data pipeline integrates multiple technologies: Kafka for data collection, Flink for stream processing, Paimon for real-time warehousing with LSM tree structure, Airflow for workflow orchestration, Spark for batch processing, and StarRocks for real-time analytics. Paimon enables fast real-time upserts through its LSM tree design, while Iceberg optimizes batch analysis with high read/write performance. The architecture separates real-time streaming and batch processing workloads, enabling end-to-end data flow from collection to BI visualization.

Click to watch →

Deep Dive

Multi-Task and Multi-Objective Learning: Studying how to train models to optimize for multiple conflicting targets simultaneously, such as click-through rate, watch time, and user engagement.

KDD 2020 Marketplace tutorial: Part 3b

KDD 2020 Marketplace tutorial: Part 3b

186

Multi-task learning traditionally combines task losses using weighted sums with pre-specified λ coefficients, but this approach faces the lambda balancing problem—assigning too much weight to one task overshadows others. Treating multi-task learning as a multi-objective optimization problem offers advantages: casting it as vector-valued loss minimization and applying KKT conditions finds descent directions improving multiple objectives simultaneously. This leads to Pareto stationary solutions or directions moving toward the Pareto optimal front, mathematically equivalent to finding the minimum norm solution in the convex hull of all solutions. This guarantees descent directions improving all tasks together, unlike fixed-weight combinations which may not yield Pareto optimal results.

Click to watch →

Han Zhao - Revisiting Scalarization in Multi-Task Learning (TSVP Talk at OIST)

Han Zhao - Revisiting Scalarization in Multi-Task Learning (TSVP Talk at OIST)

136

When solving multiple tasks simultaneously, multitask learning becomes a multi-objective optimization problem with K loss functions (one per task). Unlike single-objective optimization, multi-objective optimization cannot directly compare solutions because there is no total ordering between vectors of losses. The goal is to find Pareto-optimal solutions where no other solution improves all objectives simultaneously. This requires techniques to reduce multiple objectives to a single scalar objective for optimization.

Click to watch →

Stanford CS330 Deep Multi-Task & Meta Learning - What is multi-task learning? I 2022 I Lecture 1

Stanford CS330 Deep Multi-Task & Meta Learning - What is multi-task learning? I 2022 I Lecture 1

82.5K

Multi-task learning enables machine learning systems to learn multiple related tasks simultaneously by leveraging shared representations, while meta-learning (or learning to learn) trains systems to quickly adapt to new tasks with minimal data by learning from experience across many tasks; these approaches address critical limitations of traditional specialist systems that require extensive data and training for each individual task, making them particularly valuable in data-scarce scenarios like robotics, medical imaging, and personalized education.

Click to watch →

2A - Concentration Inequalities (Mérieux Tue 10:30)

2A - Concentration Inequalities (Mérieux Tue 10:30)

116

Multi-objective learning is defined for a given sequence of cost matrices and a target point Z in [0,1]^R. A W-Z learner is a natural generalization of PAC learning where, for any realizable distribution, when given enough training points for each of the R coordinates, the expected error is bounded by Z_i for each coordinate i. When R=1 and Z=z, this recovers standard PAC learning. When Z is replaced by 1/2 - gamma, this recovers the classic notion of weak learning. More interesting learners can be defined, such as tracking false positive and false negative errors separately in two coordinates.

Click to watch →

CS 152 NN—16: Multi task Learning

CS 152 NN—16: Multi task Learning

6.2K

Multitask learning is a regularization technique where a single neural network learns multiple tasks simultaneously using a shared body with multiple output heads. Each head produces different types of outputs (classification, regression, etc.). This approach offers two key benefits: better performance than separate networks due to reduced total weights, and improved generalization because the shared body must satisfy multiple tasks simultaneously, forcing it to learn more general features rather than overfitting to any single task's peculiarities.

Click to watch →

Deep Dive

MLOps and Scale Deployment: Deep diving into online evaluation, A/B testing design, model drift detection, and scaling model inference pipelines for billions of daily active users.

ML Ops at Reasonable Scale feat. Jacopo Tagliabue | Stanford MLSys Seminar Episode 35

ML Ops at Reasonable Scale feat. Jacopo Tagliabue | Stanford MLSys Seminar Episode 35

6.4K

At reasonable scale (companies with limited computing resources, small teams, and moderate revenue), effective MLOps can be achieved by prioritizing data quality over model complexity, leveraging Platform-as-a-Service tools to abstract infrastructure, and using modern orchestration frameworks like Metaflow to streamline ML pipelines from raw data to production deployment without requiring distributed computing expertise.

Click to watch →

MLOps Crash Course for Beginners - Part 2 | Deploy, Monitor, and Scale Machine Learning Models

MLOps Crash Course for Beginners - Part 2 | Deploy, Monitor, and Scale Machine Learning Models

580

This video covers the essential MLOps practices for deploying, monitoring, and scaling machine learning models in production environments. Key topics include three deployment strategies (batch for scheduled processing, online for instant predictions, and streaming for continuous data flows), model serving tools like MLflow Serve and TensorFlow Serving, cloud deployment on AWS/GCP/Azure, container orchestration with Docker and Kubernetes, performance monitoring with metrics tracking and drift detection, automated retraining pipelines using Apache Airflow, model versioning and feature stores for consistency, and resource optimization through caching and scaling techniques.

Click to watch →

What is ML Ops?

What is ML Ops?

76.5K

MLOps (Machine Learning Operations) streamlines the end-to-end ML lifecycle through three stages: Develop, Deploy, and Monitor. The Develop stage includes version control (GitHub/Git) for collaborative code management, data version control for dataset tracking, experiment tracking (MLflow) for reproducibility, and API development using frameworks like FastAPI. Deployment involves containerization with Docker to package models with dependencies, enabling consistent execution across environments. CI/CD pipelines automate testing, evaluation, and deployment, enforcing quality gates and enabling rapid, reliable model delivery.

Click to watch →

How he cracked 50LPA Remote Job from TCS 3.5LPA | From support engineer to MLOPs

How he cracked 50LPA Remote Job from TCS 3.5LPA | From support engineer to MLOPs

122.1K

MLOps (Machine Learning Operations) is the deployment and operational layer that bridges model development and production. Using a bakery analogy: data scientists create the cake (model), while MLOps engineers handle packaging, branding, and sales (deployment). Without MLOps, models remain limited to the developer's system. MLOps engineers implement CI/CD pipelines, deploy models on servers or serverless architectures, and perform monitoring. This role makes professionals more valuable by reducing dependencies on other teams and enabling models to be accessed by users in production environments.

Click to watch →

Interaction Session 2 || IntelleQAcademy

Interaction Session 2 || IntelleQAcademy

152

MLOps (Machine Learning Operations) addresses the Jupyter notebook problem where models are useless if software engineers cannot communicate with them. The program covers: (1) Versioning data and models; (2) Experiment tracking using MLflow; (3) Serving models by wrapping them into high-performance REST APIs using FastAPI; (4) Using Docker to ensure models run identically on any server; (5) Setting up CI/CD pipelines for automated cloud deployment. This transforms learners from standard data scientists to machine learning engineers.

Click to watch →

Problem Setup

Architecture

Label Design

Feature Eng

Model Design

Evaluation

Deployment

Problem Setup

0:00

Playing Section

1
Design a YouTube recommendation feed returning hundreds of ranked videos.
2
Must balance scale, latency, and accuracy using a multi-stage approach.
3
Define engagement labels and assume offline ingestion for new content.

46.3K views1.8K33:12@MLEpathOriginal Release: 2024-12-07

For large-scale recommendation systems like YouTube's feed, a multi-stage architecture is essential to balance accuracy, latency, and scale constraints. The system employs two main stages: a candidate generation stage that prioritizes scale and latency by filtering billions of items down to thousands using approximate nearest neighbors and embeddings, followed by a ranker stage that prioritizes accuracy on the smaller candidate pool. Labels are derived from user interactions (likes, watch time, end-screen clicks) with careful handling of class imbalance through temporal splits and negative sampling. Real-time updates use lightweight models to adapt to immediate user intent while maintaining long-term embedding stability through periodic retraining. Deployment follows AB testing with continuous monitoring and retraining every few hours to maintain model freshness.

[00:00:00]wait a minute no way you can do all that in like 30 minutes challenge accepted no script no multiple takes which is going to go beginning till end me solving one question in 30 minutes for ML system design interview so the question we're going to tackle today is design a feed for YouTube recommendations YouTube has billions of users daily and it doesn't matter if it's 500 million or three billion and you know each one of them most of them probably don't post but they're probably a good 100 million or more uh order of magnitude New pieces of content posted to YouTube every day we're going to ignore the shorts for the purposes of this uh and just stay with long uh long form content so we got a boatload of those also the recommendations on you YouTube unlike some other social networks can go back quite a while it's still relevant to me to get recommended things from a few years back and so yeah there's there's going to be some cut off we can think about it in terms of exponential decay as it gets further and further away and we could model that or we can just say we're going to cut it off at a certain point in time what I need to predict at the end of the day is what I need to produce rather is a feed of about let's say a couple of hundred videos that have the first one of which is probably the most relevant and then they decrease in relevancy a little bit but still remain pretty relevant to the user does that sound reasonable okay thank you Mr interviewer or Mrs interviewer okay so if that's what I'm predicting I I would need to know so I know the scale now but the the other thing I would like to know is uh the latency so there is this property of the recommender systems that or really anyl models that you know you could have latency you could have so by that I mean tight latency um you could have scale and you can have accuracy and any two of the three you're welcome to have but I think the problem here is that this this is business critical so we have to have accuracy latency needs to be quite tight because if you've ever sat in front of your computer for a couple of seconds while the page loaded that's not acceptable it's got to be subsecond probably 100 milliseconds or so again order of magnitude and then scale we have a tremendous scale so what I want to do do is I want to rank all the posts but unfortunately I can't do that easily here so we're going to need to talk a little bit about how to handle that the other thing is the post I'm just going to assume the actual item I guess is a video some description maybe a thumbnail and a title so let's just for Simplicity assume that th those four pieces make up an item and then let's assume that we have a system in place today because somehow we are going concerned so there definitely is some system in place even if it's just random but we can compare against that to see how well we're doing and what we're after his engagement right so likes I'm going to say that having watched maybe 30% of the video is maybe um an engagement point maybe we have a couple of those I don't really want that to be a continuous VAR able I would rather have it be watch you know 10% watch 30% watch 50% watch the whole thing something like that all right I think we're ready to jump into the rest of this uh does this sound like I understand the question are there any nuances that I might have missed okay thank you very much so I just want to go through and let you know kind of how I'm going to think this through to give you a mental mind map of what h is going to happen and just to kind of align on this right so first I want to go through big picture architecture of this and then we're going to go into some data considerations followed by modeling if only a good spell and Then followed by deployment so deployment to me is everything that happens after you have a model that you're quite comfortable with so if that's okay with you let's proceed and uh uh yeah I'll cover things in this order so for big picture what I really want to do is I want to make sure that we're covering this problem that I've identified here right so this guy right here is going to give me some problems here because I have kind of the need for all three so we're probably not going to solve all three of them in a single model and that's okay so oops I'm going to get over to the side here so a user submits a request for this feed I have some service here the service will talk to the service will talk to uh the machine learning model and and what it's probably going to first ask for is it's going to talk to service uh and this is going to be our candidate generation model so what I want to do in order to solve the triangle dilemma that I showed is I want to split this up into multiple stages so the candidate generation stage in particular really only cares about the scale and the latency we are going to have some false positives that this identifies and that's okay uh we're going to optimize it to identify as many true positives as we can and we're going to go from you know maybe a billion or like so like they're billions of posts uh I'll I'll draw that in a minute to maybe a couple of so like the input into this is maybe a billion posts again if only I could spell and the output of this is maybe a few thousand maybe a thousand I don't know order of magnitude right and then I can have another stage which is my ranker so this is going to this is what's going to determine the relative relevance to uh the user and this ranker is going to go from thousands to thousands because that's a ranker rankers don't kick things out but yeah so here we have a Rinker but this thousand is now arranged in order and then I'm going to have some non a m system I'm just going to represent it with the oval for no real reason and so this is like business rules so this is you know the most basic version of this is we can just cut off that thousand down to a couple hundred that we need uh but we probably want to take care of diversity like if all of them are videos from me um that's terrific we should keep that but if all of them are videos from I don't know somebody else then we only need like one video from that Creator um in the feed and uh yeah so business rules can also enforce a couple of other things here um and actually now that I think about it we probably want to have business rules kind of everywhere so even here we might want to start enforcing some diversity maybe not uh but it's good to have some business rules here and here we can uh sorry right here we could have business rules for like let's say the user is 18 years old or younger we don't want to surface them content that's like 21 plus let's say the user has never watched a YouTube video in Japanese we want to maybe not serve them content in Japanese right so this this just helps us basically things that we already know we can eliminate and uh for anything else you know we pass that to the candidate generation so there is another path here and there's another type of user or another user who generates the post and so user generates a post I apologize uh so post that also goes through some sort of service but um we don't care about that that much right now what matters is that the moment the post is generated I don't want to take up any of my precious time once we get to the scandid generation St stage I know or post as a video in this case I guess uh so I don't want to take any of my precious time here I want to encoded and store it as an embedding here uh at generation time I guess one thing I didn't ask that I uh didn't clarify that I probably should have is I'm assuming that it's okay that a video is not recommended right away that there's some encoding step here on YouTube it takes a little minute uh for it to go live is that a reasonable assumption okay so yeah so this gets stored as an embedding and it's from those embeddings that we start our our um exploration here the user also has activity that also goes into offline path um it could be either user but um well you know let's not do that let's uh let's do another user and now like there is activity right so like interacting with interacting with different videos like doing all sorts of stuff and I think we're going to store user embedding as well so both of these would live like in an embedding store or basically Vector store so there's some Vector store here and these are not static things so maybe we recalculate them every once in a while maybe once a day or so I'll talk a little bit more about how we deal with them in real time but overall I think for the system we do that we just recalculate them every once in a while on a Cron job and then I'm going to leave kind of that part of the system right here it's in the real time portion so this separates the real time below it from [Music] the from things that happened uh before the feed was recommended we'll talk about this box when I talk about data and this box is meant to like it's some sort of a so like real time update the idea here is that if I go on YouTube specifically to find a way to fix my car and it's 1998 Toyota Corolla or something and I watch a bunch of those videos uh the algorithm should probably adapt quicker than we can retrain these embeddings and so we want to make sure that we capture that signal so there's a there's something we need to do there but but we'll get there in a minute and then from those business rules in the end we basically generate about a th000 posts oops and those posts go back to the service and the service communicates back to the user so yeah there's not a way to do a two-way Arrow here sadly so goes back to the user okay so and this round trip should be really fast so again in the first stage in the candidate generation stage we're prioritizing uh for scale and latency and in the second stage we're prioritizing for accuracy and latency but at that point the scale is not too big just a few thousand and here we can say you know we're getting hundreds back to the user yeah how does that look is that um is that similar to what you're thinking okay good good okay so the good news is we have two of these embeddings calculated entirely offline to the system so they don't lead into our latency budget at all so let's talk a little bit about data considerations so the first thing that I want to talk about with data considerations is um our labels U so obviously hard to do machine learning without any labels so where do our labels come from well we already talked about how we have a system currently and we talked about how interaction is what we're trying to prioritize and so if interaction is what we're trying to prioritize these these could be essentially likes if you don't know about likes on YouTube take a look at the button below it has a thumb up and uh if you're still here at this point why not put a like on this video might help out the algorithm maybe watch time right so we talked about that bucket it there there are probably a few other signals like when you get to an end screen did you click on the thing on the end screen are you still watching that Creator did you click a follow maybe um which like if you're not following this awesome mle path guy you should be yeah so there's a whole bunch of signals here and I would probably do some classification problem here so this is binary multi-label classification the reason I like to have multi-label so I probably would train uh for each of those labels and then I would train overall for engagement somehow but even training for each of those labels would already give me a lot and if my product manager comes to me tomorrow and says you know what what we've decided is that we really need to prioritize likes then I can very easily tune the system um so those are my labels so those are my positive la labels my negative labels are a little bit trickier here so the trick is that you know obviously the really bad answer would be to say anything that the user hasn't clicked on they don't like well that's not exactly true because they weren't even they didn't even see most of that content so a little bit better is to say anything that appeared in their feed but they didn't click on but I think better yet is if we go through the chronological feed so one two three four right like let's say we have a very simple feed with four videos suggested the user clicked on one that's a clear positive sign the user didn't click on two we're going to count that as a negative the user clicked on three so we're going to count that as a positive the user didn't click on four or anything below that well that means I don't know anything about number four and so I probably wouldn't include that into my labels at all or I would mark it as a separate class and uh not mark it as a negative so the downside to that is we're clearly operating with a lot less data the upside is that the data that we're operating on is a lot better the other upside is that this is a highly imbalanced problem even if I only go off of this this is still probably imbalanced at least 10 to one because most of the things on your feed you're never going to click on but 10 to1 imbalance is a lot easier to deal with in machine learning than like a billion to one imbalance so if there are billions and billions of pieces of content you've only interacted with 10 of them that is actually pretty hard to deal with so we could do some subsampling there or some negative sampling or whatever but we might not even have to do that if we go off of a strategy where we only go to the last piece of content you interact with the other consideration here is that the YouTube feed is a square and so there's you know video Here video Here video Here video here and then underneath and so we need to be a little bit culturally aware here for most western cultures I believe the pattern is left to right and then uh up to down and uh but but we have to be aware of like languages like Japanese and Hebrew are read differently and because of that the scanning patterns in those in the cultures that speak those languages uh tend to be a little bit different so that's something we have to keep an eye on so those are my labels now for features the main question is where do we get features and you know here clearly you know videos we have users and we have interactions between them now in a system like this you know first of all you'd probably have 10 20 50 stages I don't you probably won't have 50 but you'd probably have you know 10 mL stages and notice how when I talked about big picture diagram I really focused in on two I think you know for the purposes of time that's probably the best use of our time and also it allows us to demonstrate some Concepts that would be useful to anything else likewise here let's just talk about a couple of features for each section rather than going into a lot of depth does that sound good okay great so so for videos obviously we already have our um you know the four things that I mentioned you know they might have tags and they might have uh like number of likes and number of likes you know you can't do in absolute terms because a video that's been alive for 30 seconds is probably going to get way fewer likes than the same video that's been alive for 100 hours so likes at some time at one hour or whatever the problem with having it be at 1 hour is that EV initially you have to train that embedding was with a nun and then you have to retrain it but that's life that's how we live so uh for tags there are way too many of them to do one hot en coding so I would probably do something like you know embeddings are pretty good and and here you could probably use off-the-shelf embedding or we might want to train our own because hashtags are sometimes a different beast from general purpose language so so probably train our own embedding for the number of likes uh at a certain time so we have two problems here one is while likes are an integer and we can't theoretically put an integer into our model there's a pretty wide distribution of what they could be so I don't want to deal too much with the outlier so I want to standardize it somehow now how to standardize it you could standardize it in a naive Way by shrinking everything down to negative 1 to one there's really no magic to that though right like what we're trying to do is we're trying to just make sure that things are basically on the same scale and for different creators different numbers of likes mean different things so for a Creator like me 500 likes on this video would be amazing for a Creator like Mark Rober 500 likes would be pretty disappointing on any video honestly and so let's maybe standardize this by the number of uh likes and average video gets or the number of followers or the average number of view or the number of views on that video something like that it won't quite be up to one it might get into the tens even that's okay I'm not too worried about it getting into tens I don't want it to get into billions too fast so uh for users you know we can have several features here so age is interesting maybe um and then you know maybe for users we also have a graph of um or we don't even have to have a full graph just like probably first connections uh there's no of connection on YouTube so like follows maybe um and uh this I would also probably encode in an embedding of some sort for age I would bucke tize it uh and then I would represent it with one hot encoding the reason for that is we're in the business of selling ads and ads actually have very strong like age demographics so you usually sell ads to like 18 to 24 year olds or whatever and so there's a finite number of buckets I'm not too worried about uh sparsity so you know we could we could do that and then interaction between them you know this is where a lot of our labels are going to come from so I don't know that um I'm going to cover that much more and then once we have all this data we need to split it somehow and uh I'm going to propose that we do a train Dev test split the reason you need that Dev set in particular is because you don't want to overfit on your uh test set so on the train set you train the model on the dev set you tune hyperparameters and on the test set you only do the evaluation before the model launches to production yeah and uh and for the split itself since the seasonality is uh probably weekly the content that you look for on weekends could be quite different from the content you look for during the week I would say that we need at least two weeks in uh both Dev and uh the evaluation set the test set so that if something is H something funky is happening on Monday we can go and check the other Monday uh and make sure that that's where it's coming from now downside to that is that you miss out on about a month of data in your training set but you know the moment that your model hits production is going to see data it's never seen before anyway uh so you need to be robust to that and uh the reason we need a temp split here is because certain things Trend based on news but then when the news dies down the the kinds of people it attracts are different and also the patterns of interaction are quite different so yeah temporal split here is quite useful and then I just want to go back here real quick uh and talk about kind of in transitioning to the modeling section the real time update piece but before then do you have any questions on what I just cover covered okay great I have a great interviewer they have no questions today it's almost like talking to you or myself all right so for the real time Updates this again is there to solve the problem of I came on the platform to solve a particular thing and so rather than generating all the candidates you now really should be shifting towards whatever I came there for and this is cashed for a short amount of time and so the way that we can do this is we can have a basic MLP that we train with uh the input is like the embedding so for the user it could be the user embedding for the post it could be the post embedding what changed so a vector of what changed and uh the output is a new embedding so basically it's a slightly modified embedding that we had in the first place so it's sequence to sequence learning a basic MLP would do this quite fast I'm not too worried about latency and then we throw it away at the end of the session uh and then next time we just read back from the initial embeddings yeah so that's what I propose and then we retrain those embeddings once a day or on a slightly longer time scale so that you know the one time you went in there to look at washing machines you didn't just ruin your feed forever with washing machines so okay so for modeling uh we basically already have the outline of the two Tower model we have the user Tower and the so two towers uh the user Tower and the uh video Tower the item Tower whatever you want to call it so basically when we have these embeddings right here uh what we do is we take a DOT product or somehow get them into the same space we find a distance measure so this needs to be the distance metric needs to be quite simple because this is something that we want to calculate fast so even cosine similarity would would probably be fine but I don't know like some distance some distance metric that's pretty easy to compute and uh what we do is we look for the videos that are closest to the user in that space so initially we look for like a couple thousand of them for the candidate generation so we can pass them on and then in the Rinker we use something similar but first let me just go back and say how we're going to look for similar things so approximate nearest neighbors is what I would pick we could do nearest neighbors but that takes up more time more resources and uh it's you know we need to go fast here so approximate nearest neighbors is the right uh algorithm here whereas in the ranker you know I would like to use the same basic philosophy and uh and and just do embeddings plus plus Ann but I would probably do something like this real-time update piece here but instead of real-time update I would give it more features so this embedding could be quite simple and then here the embedding could be a little bit more complex so we enrich that embedding and then do the same thing and what that would do is it would also allow us to do distillation back into our candidate generation model so we can train our ranker which is going to be the more advanced model first and then distill the learnings into the candidate generation model this is a class classifier essentially and uh so um and the most important thing is not to miss any true positives so what we're going to do is we're going to pick recall for our metric here and for a loss you know there's there's a modum of imbalance uh here uh I think a log based something a log loss normalized cross entropy maybe normalized cross entropy would deal with the imbalance a little bit better so let's put normalized cross entropy both here and here as our loss metric and then on the Rinker itself we can't use a metric like recall I mean we could but it wouldn't be applicable um so instead I'm choosing between Mr and ndcg Mr is better when you're trying to guess the first answer right and you don't really care about the order beyond that the since we're trying to get an entire feed that's basically correct ndcg is my metric of choice here so that's what I would start with n dcg and uh recall yeah and obviously like if anything goes wrong you monitor them and you adjust but those are starting points here yeah any questions on modeling okay yeah thank you yeah so that gets us a model we have our train and test splits we can yeah and if we train the ranker first and then go for candidate generation then we can use distillation to be a lot more resource efficient okay so how are we going to launch that to production so for production we need to run an AB test we need to Canary deploy this because other deployment possibilities such as uh blue green deployment for example require us to stand up a whole other system to deploy alongside ours and at these scales is just C prohibitive so AB test here works well we don't really have anything that like lives depend on it if we did if it was like unethical to make the wrong choice for example then I would recommend Bandits instead because Bandits can converge to a solution and then start recommending more content from your B system uh than a if B is truly better but here we have time to run an entire AB test and um honestly yeah I think that's as good a solution as any uh we do need to stratify that by users so the same user has to be either in the a group or the B group cannot be in both and then we run that for a while we need to get some statistics of the like the magnitude of the effect versus how long we ran it and on how many users we measure online metrics which you know you could go with the same things you could go with engagement metrics as online metrics you could also go with just how long users spend on the platform or the number of ads you can show or something like that number of ads is probably a little bit too abstract so let's let's stick to the same engagement metrics but yeah time on platform is also possible here and then once we deploy it we need to make sure we retrain it every once in a while so for a model like this you know content is uploaded all the time things are happening all the time so maybe every few hours I would retrain it and monitor normalized entropy and as long as that's trending in the right direction or at least staying constant I would be launching newer and newer models every four hours automatically and if not then you know the first time I'd probably let a go the second time i' probably get alerted on the on call speaking of which uh I would probably monitor the inputs and outputs to every stage so every ml model even this one needs to have inputs and outputs monitored outputs are usually my fault as an mle uh inputs are usually the data team's fault but I still I'm not so much caring about assigning blame but I need to know what's happening to my model in production and I will alert the right on calls uh after that so so yeah so uh we can do that and that's basically it and I did not make it in 30 minutes which is okay you actually have about 37 minutes uh or more even like 40 sometimes uh to cover everything in this interview my interviewer was incredibly kind to me and didn't really have a lot of questions uh to direct back in me but I feel like as long as you keep the interview moving and as long as you give your interviewer a perception that you'll get there at least right like so the reason I outlined early what I'm going to say and when is because I don't want my interviewer in this stage being like okay like what is the model here or what is the model here I want to make sure that that is covered later and that their brain is at rest test that it will be covered later also remember that you don't need to hit every single point here you just need to make sure that you give your interviewer enough do watch the ml system design video it'll really help you solidify the ideas I talked about here

Up Next

HNSW Vector Search: Implementation with Faiss in Python

@jamesbriggs

50.3K views•2021-10-05

BitTorrent Protocol Explained: Piece Selection & Peer Choking

@StevenGordonAU

481 views•2013-02-22

HTTP Requests Explained: GET, POST, PUT, DELETE

@codecademy

103.1K views•2021-10-07

Enigma Machine Mechanics: WWII Encryption Explained

@JaredOwen

13.2M views•2021-12-11

Related Study Plans & Knowledge Roadmaps

Structured learning paths in Computer Science

View All Computer Science Topics

computer-science

Building BitTorrent: Go, P2P & Concurrency

01Go Fundamentals & Concurrency

02Networking Foundations & P2P Architecture

03Bencode Parsing & Torrent Metadata

+ 3 more learning modules

Explore Roadmap

computer-science

Actor Models (C++ Concurrency & Lock-Free)

01C++ Multi-Threading Foundations

02Thread Pools and Task Scheduling

03Atomics and Lock-Free Queue Design

+ 3 more learning modules

Explore Roadmap

computer-science

Coding HM Type Inference: Algorithm W & OCaml

01Introduction to Functional Programming & OCaml

02Abstract Syntax Trees & Language Interpreters

03Type Systems & Lambda Calculus

+ 2 more learning modules

Explore Roadmap

computer-science

Building OAuth 2.0: Node.js OIDC Server

01Web & Security Foundations

02Authentication vs. Authorization & JWTs

03The OAuth 2.0 Protocol & Grant Types

+ 3 more learning modules

Explore Roadmap