Academic Databases
Explore Our Semantic Databases
STEM & Tech
Math, Physics, CS & Engineering...
Economics
Macro, Micro, Econometrics...
Business
Management, Marketing, Finance...
Health & Life Sciences
Biology, Medicine, Neuroscience...
Humanities & Social Sciences
Psychology, Law, Philosophy, History...
General Studies
General & interdisciplinary studies...
Explore Topics
Features

Multivariate Time Series Preprocessing with Pandas in Python

Added: 2026-08-07

Deep Dive

Fundamental Python programming and core Pandas operations, such as DataFrame manipulation, indexing, and handling missing data.

Complete Pandas in 2 Hours + Free Notes | One Shot Series (Hindi)

Complete Pandas in 2 Hours + Free Notes | One Shot Series (Hindi)

32.9K

Pandas is an open-source Python library for data manipulation and analysis, freely available to all users. Install with pip install pandas or Anaconda Navigator. Import with: import pandas as pd. Check version with: pd.__version__. Pandas has two fundamental data structures: Series (single-column table) and DataFrame (multi-column table). Create Series with: pd.Series([data], name='name'). Access elements by index: series[0]. Use slicing: series[start:end]. Methods include head(n), tail(n), sort_values(), count(), isnull(), notnull(), and fillna(value). Create DataFrames with: pd.DataFrame({'column1': [values1], 'column2': [values2]}). Access values with: df.iloc[row, col] (positional) or df.loc[row, col] (label-based). Methods include head(n), tail(n), info(), and describe(). Add columns with: df['new_column'] = [values]. Remove columns with: df.drop('column', axis=1, inplace=True). Rename columns with: df.rename(columns={'old': 'new'}, inplace=True). Set index with: df.set_index('column', inplace=True). Reset index with: df.reset_index(inplace=True).

Click to watch →

30dk'da Python Pandas Eğitimi

30dk'da Python Pandas Eğitimi

18.8K

Pandas is a Python library for data manipulation that enables users to create, read, and analyze tabular data through DataFrames; key operations include importing libraries (import pandas as pd), reading data from files (read_csv, read_excel), filtering data using boolean conditions (df[df.column == value]), selecting rows and columns (df.head(), df.tail(), df.loc, df.iloc), performing calculations on columns (df.column * 3), converting data types (astype), handling missing values (dropna), and exporting data to Excel (to_excel).

Click to watch →

Series and DataFrame in 1 Video | Data Handling using pandas | CBSE Class 12 Informatics Practices

Series and DataFrame in 1 Video | Data Handling using pandas | CBSE Class 12 Informatics Practices

16.1K

This comprehensive section covers Pandas fundamentals including: Python libraries (NumPy, Pandas, Matplotlib) and their purposes, data analysis concepts, Pandas features, installation process, and the critical difference between Series (1D, homogeneous) and DataFrame (2D, heterogeneous). It details Series creation from various data types (lists, tuples, strings, dictionaries, range functions, scalar values, NumPy arrays), Series function arguments (data, index, dtype), custom index specification, and representing missing data using NaN values. The section covers Series indexing methods (label and positional), slicing operations with start:stop:step syntax, Series attributes (index, values, dtype, shape, size, itemsize, nbytes, ndim, hasnans, empty, name), and functions (head(), tail(), count()). It then covers element-wise mathematical operations (addition, subtraction, multiplication, division) and vector operations, explaining how operations work when indices match and when they don't (NaN for missing values). The section concludes with Series filtering using boolean conditions, sorting values and indices, and transitions to DataFrame creation methods (empty DataFrame, NumPy arrays, list of dictionaries, dictionary of lists, dictionary of Series, dictionary of dictionaries).

Click to watch →

Conviértete en el mejor Científico de Datos 🙌🏼 y aprende Pandas

Conviértete en el mejor Científico de Datos 🙌🏼 y aprende Pandas

54.4K

Data analysis is a science for obtaining, classifying, analyzing, and drawing conclusions from databases, with applications in AI, industry, healthcare, and entertainment. Pandas is a Python library for inspecting, cleaning, and transforming data, importing from databases and Excel. Series are numbered lists with labels (index), created using pd.Series(). Functions like head(), tail(), min(), max(), and mean() explore data. Access elements using square brackets or get(). Filtering applies conditions to create boolean Series. Mathematical operations are element-wise, matching by index rather than position. DataFrames are tabular structures created from dictionaries, with keys as column names and values as column data. The index is automatically assigned as consecutive numbers. Functions like head() and tail() work similarly to Series. Selecting columns uses the columns parameter. Using a column as the index requires the index parameter. The loc() function selects data by label, while iloc() selects by integer position. Filtering uses conditions: df.loc[df['surface_km2'] > 1000000] selects rows meeting criteria. Modifying values uses loc() assignment.

Click to watch →

Class 12 Python Pandas | Adding, Deleting, Updating, Renaming Rows Columns Cells in DataFrame Object

Class 12 Python Pandas | Adding, Deleting, Updating, Renaming Rows Columns Cells in DataFrame Object

45.3K

This tutorial covers essential Pandas DataFrame operations for data manipulation: accessing individual elements using column names and indices (e.g., df['column'][index]), adding new columns by assigning lists of values (df['new_column'] = [values]), modifying existing columns by replacing values (df['column'] = [new_values]), updating single cells (df['column'][index] = value), deleting columns using del keyword (del df['column']), renaming rows and columns using index and columns properties (df.index = [new_names], df.columns = [new_names]), and using boolean values (0/1) for on/off states in data.

Click to watch →

Deep Dive

Basic concepts of time series data, including datetime indexing, frequency, and the difference between univariate and multivariate datasets.

Введение в ML, временные ряды (часть 1)

Введение в ML, временные ряды (часть 1)

1.1K

A time series is a sequence of measurements or realizations of random variables recorded at different time points. Time series can be univariate (single component) or multivariate (multiple components). The distinction between univariate and multivariate time series is crucial: univariate time series have one measurement per time point, while multivariate time series involve multiple measurements (such as x and y coordinates) at each time point. This classification affects how we analyze and model the data.

Click to watch →

Introduction to Time Series Analysis Hands-on R Software by Dr. Varun Agiwal

Introduction to Time Series Analysis Hands-on R Software by Dr. Varun Agiwal

401

Time series can be univariate (single variable observed over time) or multivariate (multiple variables observed simultaneously): (1) Univariate time series - focuses on a single variable (e.g., daily temperature); (2) Multivariate time series - examines relationships between multiple variables (e.g., temperature and humidity together). The choice depends on research objectives and data availability.

Click to watch →

Introduction to Time Series Modelling - Time Series and Stationarity concept (2022)

Introduction to Time Series Modelling - Time Series and Stationarity concept (2022)

232

Time series data is collected at regular intervals regardless of frequency (daily, weekly, hourly, or per-second). Common examples include stock prices (daily maximum, minimum, closing prices), weather data, sales data (critical for business forecasting and planning), and visitor counts. A time series structure includes a date/time column and the measured values. Univariate time series contain only one main variable, while multivariate time series include additional variables like holidays or weather conditions that help predict the main variable.

Click to watch →

Séries Temporais: Conceitos Básicos 03

Séries Temporais: Conceitos Básicos 03

1.9K

A time series is a collection of observations sequentially ordered in time, such as monthly electricity production data from 1956-1995. Time series data consists of an index (typically time) and corresponding values. Data can be collected at regular or irregular intervals, with regular intervals being preferred for most analytical techniques. The sampling frequency directly impacts data quantity, with higher frequency producing more observations. The index can be any temporal unit (day, month, hour), though many techniques replace it with sequential numbers. A key assumption is that observations are serially correlated, meaning values at one time point relate to values at other time points, distinguishing time series from independent variables.

Click to watch →

[시계열] Ch4. 시계열 데이터 분석 개념과 응용

[시계열] Ch4. 시계열 데이터 분석 개념과 응용

530

Time series are classified by variable count: Univariate tracks single variables (stock prices, temperature, sales) for pattern identification. Multivariate tracks multiple interrelated variables simultaneously (economic indicators, climate factors), revealing interactions between variables. Univariate analysis identifies single-variable trends, while multivariate analysis captures complex relationships where variables influence each other, enabling more comprehensive system understanding.

Click to watch →

Deep Dive

Core statistical preprocessing techniques, specifically data normalization and standardization (e.g., MinMax Scaling) and why they are vital for machine learning models.

Standardization Vs Normalization - Qual utilizar em tarefas de Machine Learning?

Standardization Vs Normalization - Qual utilizar em tarefas de Machine Learning?

18.6K

This comprehensive lesson covers data preprocessing in machine learning, focusing on standardization and normalization. The problem of different feature scales is explained: when features have different units and magnitudes (age in years, height in centimeters, weight in kilograms), algorithms may incorrectly assign disproportionate importance to features with larger numerical values. Normalization rescales data to a 0-1 range using (x - min) / (max - min), while standardization transforms data by subtracting the mean and dividing by standard deviation: (x - mean) / std. Practical implementation using sklearn.preprocessing is demonstrated, including StandardScaler and MinMaxScaler. The lesson shows how preprocessing dramatically improves model performance (from 75% to 97% accuracy). Guidelines for choosing techniques: Standardization is best for distance-based algorithms (KNN, SVM) and gradient descent methods, while Normalization is preferred for neural networks. Tree-based algorithms are generally unaffected by preprocessing.

Click to watch →

Statistics And Probability Tutorial | Statistics And Probability In Data Science | Intellipaat

Statistics And Probability Tutorial | Statistics And Probability In Data Science | Intellipaat

7.2K

Data preprocessing is crucial for machine learning, particularly when features have different scales and units. Standardization (z-score normalization) transforms data to have mean 0 and standard deviation 1 using z = (x - μ)/σ. It preserves data distribution shape and is essential for algorithms using gradient descent (linear regression, neural networks) and those relying on Euclidean distance (k-nearest neighbors, k-means clustering). Normalization (min-max scaling) transforms data to range 0-1 using x' = (x - min)/(max - min). It compresses outlier effects and suits algorithms sensitive to feature scales. Tree-based classifiers don't require either. Choice depends on data characteristics: standardization handles outliers better, normalization suits bounded ranges. Both enable fair comparison of features with different magnitudes.

Click to watch →

Normalización| Preprocesamiento de datos | SciKitLearn | Machine Learning| Python | ¡Muy básico!

Normalización| Preprocesamiento de datos | SciKitLearn | Machine Learning| Python | ¡Muy básico!

18.2K

Data normalization and standardization are essential preprocessing techniques in machine learning that transform features to have similar scales, improving model performance. Standardization (Z-score normalization) transforms data using the formula: (x - mean) / standard deviation, resulting in a distribution with mean 0 and standard deviation 1. This is particularly important when features have vastly different ranges, such as movie ratings (0-10) versus Facebook likes (0-349,000). Python's scikit-learn library provides StandardScaler and MinMaxScaler for automated normalization, while pandas simplifies statistical calculations like mean and standard deviation. Proper data scaling ensures that machine learning algorithms can learn patterns more effectively without being dominated by features with larger numerical values.

Click to watch →

Máy Học Ứng Dụng - Bài 20. Tiền Xử Lý Dữ Liệu Trong Máy Học - P2 Normalization and Standardization

Máy Học Ứng Dụng - Bài 20. Tiền Xử Lý Dữ Liệu Trong Máy Học - P2 Normalization and Standardization

2.7K

Data normalization and standardization are essential preprocessing techniques that scale feature values to ensure equal contribution to machine learning models. Min-Max normalization rescales data to a fixed range (typically 0-1) using the formula: (X - X_min) / (X_max - X_min), while standardization transforms data to have a mean of 0 and standard deviation of 1 using the formula: (X - mean) / standard_deviation. Min-Max normalization is suitable when data distribution is unknown or non-Gaussian, while standardization works best for Gaussian-distributed data. The choice between methods depends on the data characteristics and algorithm requirements, with normalization being more sensitive to outliers.

Click to watch →

Padronização e Normalização de Dados para Ciência de Dados

Padronização e Normalização de Dados para Ciência de Dados

5.2K

Data standardization and normalization are essential preprocessing techniques that ensure machine learning models can correctly analyze features with different scales by adjusting data to a common scale, preventing features with larger magnitudes from dominating the model's learning process. Standardization (Z-score normalization) transforms data to have a mean of 0 and standard deviation of 1 using the formula (x - mean) / std, while normalization (Min-Max scaling) rescales data to a specific range like [0, 1] using (x - min) / (max - min). For datasets with outliers, RobustScaler uses median and interquartile range instead of mean and standard deviation to avoid distortion. These techniques are crucial for improving model performance and ensuring fair feature importance assessment.

Click to watch →

Deep Dive

The concept of supervised learning representation for sequence data, specifically the sliding window approach to create inputs and targets.

LSTMs - End to End Guide | Clearly Explained | Satyajit Pattnaik

LSTMs - End to End Guide | Clearly Explained | Satyajit Pattnaik

6.3K

This section covers creating supervised learning datasets for text prediction tasks. It explains: (1) The process of sliding windows over tokenized sequences to create input-output pairs; (2) How each input sequence predicts the next word; (3) The distinction between classification (discrete word prediction) versus regression; (4) One-hot encoding for multi-class classification targets; (5) The complete workflow from raw text to structured X and y arrays. This bridges the gap between raw data and model-ready inputs.

Click to watch →

What is LSTM ?| Long Short Term Memory Explained with Example | Deep Learning Tutorial | Simplilearn

What is LSTM ?| Long Short Term Memory Explained with Example | Deep Learning Tutorial | Simplilearn

4.9K

The sliding window technique converts time series data into supervised learning format by using previous time steps as input features and the next time step as the target output. For example, with a window size of 60, each input sample contains the last 60 timestamps' prices, and the output is the price at timestamp 61. This approach transforms sequential data into a format suitable for neural network training.

Click to watch →

Mastering Time Series Forecasting: Build a Transformer Model in Keras - Predict Stock prices

Mastering Time Series Forecasting: Build a Transformer Model in Keras - Predict Stock prices

8.9K

The sliding window approach creates overlapping input-output pairs from a time series. Given a time series [x1, x2, x3, x4, x5, x6] and window size 3, the inputs become [x1,x2,x3], [x2,x3,x4], [x3,x4,x5] and the targets are x4, x5, x6 respectively. This approach generates many training examples from a single time series and mimics real-world forecasting scenarios where a fixed historical window is used to predict the next value.

Click to watch →

📚 Data Sampling with a Sliding Window – Live Coding with Sebastian Raschka (Chapter 2.6)

📚 Data Sampling with a Sliding Window – Live Coding with Sebastian Raschka (Chapter 2.6)

608

The sliding window technique prepares input-output pairs for LLM training by creating overlapping chunks of text. Given a context size (e.g., 4 tokens), the input consists of the first N tokens while the targets are those same N tokens shifted by one position. This creates pairs where the LLM learns to predict the next token. For example, with tokens [A, B, C, D, E], the first input is [A, B, C, D] with target [B, C, D, E]. The next input is [B, C, D, E] with target [C, D, E, F]. This pattern continues, enabling the LLM to learn sequential prediction. The technique transforms raw text into structured training data suitable for model optimization.

Click to watch →

ML lab12-6: RNN with Time Series Data

ML lab12-6: RNN with Time Series Data

58.6K

The sliding window approach converts raw time series data into sequences by extracting overlapping windows of length N as inputs and the next time step as targets. Data is split chronologically (70% training, 30% testing) to maintain temporal integrity. Batch size combines sequence length and input dimensions for X, while Y maintains single-value structure. LSTM cells process sequences with configurable hidden sizes (typically 10-20). A fully connected layer transforms LSTM outputs to final predictions. Loss functions like Mean Squared Error measure prediction error, and optimizers minimize this error through iterative weight adjustment.

Click to watch →

Setup & Imports

Data Loading

Feature Engineering

Data Transformation

Data Splitting

Feature Scaling

Scaling Applied

Sequence Creation

Sequence Example

Final Verification

Setup & Imports

2:01

Playing Section

1
Import libraries for data processing, visualization, and deep learning.
2
Set up plotting styles and seed for reproducibility.

Deep Dive

Building and training sequence-to-sequence deep learning models, such as Recurrent Neural Networks (RNNs), LSTMs, or GRUs, using TensorFlow/Keras or PyTorch.

TensorFlow Tutorial 6 - RNNs, GRUs, LSTMs and Bidirectionality

TensorFlow Tutorial 6 - RNNs, GRUs, LSTMs and Bidirectionality

53.7K

This tutorial demonstrates how to implement Recurrent Neural Networks (RNNs), Gated Recurrent Units (GRUs), and Long Short-Term Memory (LSTMs) in TensorFlow using the MNIST dataset. The key concepts covered include: (1) preparing image data for sequence models by unrolling 28x28 pixel images row-by-row as time steps; (2) building sequential models with SimpleRNN, GRU, or LSTM layers, specifying input shapes (None, 28) and using return_sequences=True to enable stacking multiple layers; (3) adding dense output layers for classification; (4) compiling models with appropriate loss functions (sparse_categorical_crossentropy), optimizers (Adam), and metrics; (5) training models with specified batch sizes and epochs; and (6) implementing bidirectional layers using layers.Bidirectional() to process sequences in both forward and backward directions simultaneously, which doubles the number of hidden units. The tutorial shows that GRUs and LSTMs generally outperform simple RNNs, with bidirectional approaches providing additional improvements depending on the dataset.

Click to watch →

S2 E4 Long Short-Term Memory (LSTM) Explained Simply, Intuitively and Comprehensively

S2 E4 Long Short-Term Memory (LSTM) Explained Simply, Intuitively and Comprehensively

Building an LSTM model involves: (1) Preprocessing text (extracting plain text, tokenizing, building vocabulary), (2) Converting sentences to padded sequences, (3) Creating model with embedding layer and LSTM layer, (4) Compiling with softmax output and cross-entropy loss, (5) Training for multiple epochs, (6) Generating text autoregressively: provide seed text, predict next word, append to seed, repeat. A demonstration on a news article about Modi's US visit shows the model successfully predicting continuations like 'looks forward to hosting President Trump in New Delhi for Quad leader summit ahead of which the leaders will...' despite training on limited data.

Click to watch →

TensorFlow Tutorial 10 - Recurrent Neural Nets (RNN & LSTM & GRU)

TensorFlow Tutorial 10 - Recurrent Neural Nets (RNN & LSTM & GRU)

49.9K

Recurrent Neural Networks (RNNs) are neural networks designed to handle sequential data by allowing previous outputs to be used as inputs while maintaining hidden states, making them suitable for tasks like text generation, translation, and classification. In TensorFlow/Keras, RNNs can be implemented using the Sequential API by reshaping input data into sequences (e.g., treating a 28x28 image as a sequence of 28 time steps with 28 features each), adding an RNN layer (SimpleRNN, LSTM, or GRU) with specified units, and optionally stacking multiple RNN layers by setting return_sequences=True. The return_sequences parameter controls whether the output returns all time steps (useful for stacking) or just the final output. For classification tasks, a dense output layer is added, and the loss function should specify from_logits=True if no activation is used in the final layer.

Click to watch →

Stock Prediction using Machine Learning and Python | Machine Learning Training | Edureka

Stock Prediction using Machine Learning and Python | Machine Learning Training | Edureka

222.1K

Keras is a high-level API for building and training deep learning models built on top of TensorFlow. Key components include: Sequential (linear stack of layers), Dense (regular fully connected neural network layer used to change output dimensions through matrix-vector multiplication with trainable parameters updated during backpropagation).

Click to watch →

Deep Learning with Python, TensorFlow, and Keras tutorial

Deep Learning with Python, TensorFlow, and Keras tutorial

1.4M

Keras provides a high-level API for building neural networks. The sequential model type is the most common and represents a linear stack of layers. The model is constructed using model.add() syntax. For image data, a Flatten layer converts 2D pixel grids into 1D arrays suitable for dense layers. Dense layers contain neurons with specified units (number of neurons) and activation functions. The Rectified Linear Unit (ReLU) is recommended as the default activation function for hidden layers due to its effectiveness in preventing vanishing gradients.

Click to watch →

Deep Dive

Advanced financial feature engineering, including the calculation of technical indicators (e.g., RSI, MACD, Bollinger Bands) to enrich multivariate datasets.

볼린저밴드 "이렇게" 설정하면 저점을 정확히 잡을 수 있습니다. (보는법, 단타 매매법)

볼린저밴드 "이렇게" 설정하면 저점을 정확히 잡을 수 있습니다. (보는법, 단타 매매법)

101.5K

Combining Bollinger Bands with RSI increases reliability: when RSI breaks above the middle band, uptrend is likely; when RSI breaks below, downtrend is likely. With MACD, when histogram touches upper band twice and then turns down, it signals potential reversal. When histogram touches lower band twice and turns up, uptrend may begin.

Click to watch →

CURSO DE PROGRAMAÇÃO DE ROBÔS DE DAY TRADE PARA PROFIT [TUTORIAL INICIANTE COMPLETO]

CURSO DE PROGRAMAÇÃO DE ROBÔS DE DAY TRADE PARA PROFIT [TUTORIAL INICIANTE COMPLETO]

9.2K

This section covers three popular technical indicators and how to implement them in trading code. The RSI (Relative Strength Index) measures momentum and identifies overbought/oversold conditions, requiring only a period parameter. The Stochastic Oscillator (both fast and slow versions) measures similar conditions with additional parameters for period and moving average type. The Bollinger Bands consist of three lines (middle moving average plus upper and lower bands based on standard deviation), requiring parameters for deviation, period, and moving average type. The instructor demonstrates how to switch between indicators by changing function names while keeping the same variable structure.

Click to watch →

COMPLETE BEGINNER COURSE TO CRYPTOCURRENCY TRADING; All you need to know to start trading.

COMPLETE BEGINNER COURSE TO CRYPTOCURRENCY TRADING; All you need to know to start trading.

464.2K

Three key technical indicators guide trading decisions: (1) RSI (Relative Strength Index) identifies oversold (0-30, good buying) and overbought (70-100, good selling) conditions; (2) Moving averages smooth price data to reveal trends, acting as support in uptrends and resistance in downtrends; (3) Bollinger Bands measure volatility—bands expanding indicates high volatility, contracting indicates low volatility (Bollinger Squeeze). Traders enter positions after breakouts from squeeze patterns, with bands serving as dynamic support/resistance levels depending on trend direction.

Click to watch →

Technical Analysis: Everything YOU NEED TO KNOW!!

Technical Analysis: Everything YOU NEED TO KNOW!!

1.1M

MACD (Moving Average Convergence Divergence) identifies trend changes and measures volatility. When MACD lines diverge widely, volatility is low; when they converge closely, volatility increases. Line crossovers signal impending price direction changes. RSI (Relative Strength Index) operates on a 0-100 scale: values below 30 indicate undervaluation (buy signal), values above 70 indicate overvaluation/bubble territory (sell signal). Bollinger Bands measure volatility through expanding/contracting bands; wider bands indicate higher volatility. Traders buy during low-volatility periods and sell during high-volatility periods. Price piercing through upper bands signals potential sell opportunities, while falling beneath lower bands signals potential buy opportunities. Combining moving averages, MACD, RSI, and Bollinger Bands provides comprehensive market analysis, though results vary by timeframe and different cryptocurrencies require tailored strategies.

Click to watch →

Alpaca Market Data API (Part 3) - Technical Analysis (RSI, MACD) with Backtrader bta-lib

Alpaca Market Data API (Part 3) - Technical Analysis (RSI, MACD) with Backtrader bta-lib

15.3K

This section demonstrates applying multiple technical indicators to financial data using TA-Lib. The tutorial shows how to calculate Simple Moving Averages (SMA) for trend identification, Relative Strength Index (RSI) for overbought/oversold conditions (RSI > 70 = overbought, RSI < 30 = oversold), and Moving Average Convergence Divergence (MACD) for momentum analysis. The demonstration includes filtering DataFrame rows based on indicator values to identify specific trading signals and validating results against known market events.

Click to watch →

Deep Dive

Evaluating time series forecasting models using specialized validation techniques like walk-forward validation (backtesting) to prevent data leakage.

Time Series Forecasting Using Recurrent Neural Network and Vector Autoregressive Model: When and How

Time Series Forecasting Using Recurrent Neural Network and Vector Autoregressive Model: When and How

95.5K

Walk-forward validation is the correct approach for validating time series forecasting models because random sampling violates temporal dependencies. The procedure involves: (1) Dividing the data chronologically into training and test sets, (2) Training the model on an initial portion of the data, (3) Forecasting the next period(s), (4) Adding the actual observation to the training set, (5) Repeating the process forward through the dataset. This method ensures the model is evaluated on its ability to predict future values based on historical information, maintaining the temporal structure inherent in time series data.

Click to watch →

I Tested Google’s autoBNN AI Model on EUR/USD Time Series Data

I Tested Google’s autoBNN AI Model on EUR/USD Time Series Data

38.5K

This section presents walk-forward validation as the appropriate evaluation methodology for time series forecasting models. Unlike standard cross-validation, walk-forward validation respects temporal ordering by iteratively training on earlier data and testing on subsequent periods. The process involves defining a window size (e.g., 100 candles), training on a subset (e.g., 50 candles), testing on the remainder, then sliding forward by a specified step size. This approach prevents data leakage and provides realistic performance estimates by simulating real-world deployment where models must adapt to evolving market conditions. The presenter demonstrates calculating accuracy as the proportion of correct predictions across all test windows, achieving approximately 62% accuracy on EUR/USD hourly data for predicting price movements within five candles.

Click to watch →

Predict NBA Games With Python And Machine Learning

Predict NBA Games With Python And Machine Learning

62.3K

Backtesting evaluates model performance on historical data while respecting the temporal order. For NBA prediction, you split data by season and train on past seasons to predict future ones. For example, use 2016-2017 data to predict 2018, then 2016-2018 to predict 2019, and so on. This prevents data leakage where future information would influence predictions and provides a realistic assessment of how the model would perform in practice.

Click to watch →

Weather Prediction With Python And Machine Learning [W/Code]

Weather Prediction With Python And Machine Learning [W/Code]

97.6K

This section covers proper validation techniques for time series data where standard cross-validation fails due to temporal dependencies. The video explains backtesting as a specialized validation approach that preserves chronological order by training on earlier data and testing on later data. The implementation involves defining a warm-up period (10 years), iteratively creating training sets from data up to each point, and test sets from subsequent intervals (90 days). The section demonstrates evaluating model performance using mean absolute error (MAE), calculating the average difference between predicted and actual values. An initial MAE of approximately 5°F indicates the model's average prediction error, providing interpretable insight into predictive accuracy.

Click to watch →

Predict House Prices With Machine Learning And Python [Full Tutorial]

Predict House Prices With Machine Learning And Python [Full Tutorial]

41.1K

Backtesting is essential for time series validation because standard cross-validation techniques would incorrectly use future data to predict past events. The approach involves iteratively training on earlier data and testing on subsequent periods (e.g., train on 5 years, predict next year, then expand training window). This respects temporal order while evaluating model performance across the entire dataset.

Click to watch →

Deep Dive

Deploying the preprocessing and forecasting pipeline into a real-time streaming architecture for algorithmic trading or live market monitoring.

Real-time Feature Pipelines with Quix Streams

Real-time Feature Pipelines with Quix Streams

2.3K

This comprehensive tutorial covers building and deploying production-ready real-time feature pipelines for machine learning applications, particularly trading bots. The pipeline architecture consists of three core steps: (1) Ingestion - connecting to external services like Kraken Websocket API to receive raw data in real-time, (2) Transformation - converting raw data into useful features by bucketing trades into time intervals and generating OHLC (Open, High, Low, Close) candles using stateful window operators, and (3) Storage - saving features to a feature store for ML model access. Each step is implemented as a containerized service communicating through Apache Kafka, enabling scalability through consumer groups. Quix Streams provides a cloud-native Python library combining Kafka's scalability with an easy-to-use interface. The implementation uses Streaming DataFrames for pandas-like operations on real-time data, with stateful window operators enabling complex feature engineering. Deployment options include local development with Docker Compose and production deployment via Quix Cloud, a managed Kubernetes platform. Each service requires an app.yaml configuration file defining app name, input/output topics, and environment variables.

Click to watch →

LIVE: Building an Algo Trading Framework | From 0 To Algo Trader

LIVE: Building an Algo Trading Framework | From 0 To Algo Trader

238

Building an algorithmic trading framework requires multiple interconnected services: an ingestion service for data collection, a broker service for market data, and a database for storage. The system handles data gaps from downtime through backfilling, then transitions to live updates. Real-time data processing involves forming candles from tick data, with subscriptions managed carefully to prevent premature stream termination. Services must start in correct order based on dependencies, enforced through Docker Compose. Front-end applications cannot directly access Redis streams, requiring a web server proxy. Redis stream consumers use batch size and queue size parameters to control memory usage. Unified backend languages simplify development, while proper configuration validation prevents runtime errors.

Click to watch →

Stock Price Prediction using real time data on Aws Ec2 Instance Using Kafka Pyspark MlLib Flask

Stock Price Prediction using real time data on Aws Ec2 Instance Using Kafka Pyspark MlLib Flask

205

This segment covers the complete streaming data pipeline implementation. First, start Zookeeper server using 'bin/zookeeper-server-start.sh config/zookeeper.properties' for cluster coordination. Then start Kafka broker using 'bin/kafka-server-start.sh config/server.properties' for message storage. Run the producer script 'python3 producer.py' to send stock data from Alpha Vantage API to Kafka topics. Transfer the stock data model file and run the main application 'python3 app.py' to process streaming data with Spark MLlib. Finally, access visualization at the public DNS address with port 8080, displaying predicted (blue line) versus actual (green lines) stock prices in real-time.

Click to watch →

Predicting Cryptocurrency Exchange Rates by Tim Frey

Predicting Cryptocurrency Exchange Rates by Tim Frey

449

The speaker describes building an engine for real-time forecasting. They train different algorithms in a stream processor deployed dynamically, creating connector microservices that talk to different exchanges. They buffer data in case of loss and handle data filtering. They also run different revisions of microservices simultaneously to test new services while keeping old ones running for comparison. The architecture shifted from Lambda (everything in a data lake) to Kappa (everything as a stream reacting to events). This required separate component deployment and introduced challenges like downtime and network connectivity issues.

Click to watch →

Dagster Deep Dive - Dagster Running Dagster - Event Driven Pipelines

Dagster Deep Dive - Dagster Running Dagster - Event Driven Pipelines

1K

This webinar demonstrates how to implement event-driven pipelines using Dagster for orchestration, Kafka for real-time event streaming, and Flink for stream processing, replacing traditional batch polling approaches with a scalable architecture that lands processed data in S3 as partitioned Parquet files, then uses Dagster sensors to trigger downstream processing in Snowflake with automatic alerting for pipeline health monitoring.

Click to watch →

Setup & Imports

Data Loading

Feature Engineering

Data Transformation

Data Splitting

Feature Scaling

Scaling Applied

Sequence Creation

Sequence Example

Final Verification

Setup & Imports

2:01

Playing Section

1
Import libraries for data processing, visualization, and deep learning.
2
Set up plotting styles and seed for reproducibility.

37.4K views83930:24@venelin_valkovOriginal Release: 2021-03-27

This tutorial demonstrates how to preprocess multivariate time series data for machine learning models, covering key steps including loading and cleaning data, creating lag features using pandas.shift(), generating additional features like day-of-week and price changes, splitting data into training and test sets while maintaining chronological order, applying min-max scaling to normalize features, and converting the processed data into sequences suitable for neural network training.

[00:00:00]hey guys in this video we are going to see how we can do some time series data pre-processing and we are going to convert a pandas data frame into a set of sequences which are going to be used in later videos for a time series forecasting classification and probably anomaly detection in our case we are going to convert some multivariate data into time series sequences and the data is going to basically contain multiple features so i'm going to show you some of the tricks that i use to convert uh this kind of data into sequences which we are going to later on convert into pytorch datasets and i'm going to show how you can train time series data or some time series models that are based on neural networks of course using pytorch lightning so stay tuned for the next couple of videos as well let's get started and i'll show you how you can do some time series data pre-processing so here we have an empty notebook on google co-op and i'm going to start by doing some library installations i'm going to connect the notebook we don't really need any gpu at least for now and i'm going to install a specific version of tqgm this will basically allow us to do some progress visualization and something cool that i've noticed about this google co-op instances at least recently is that we have those execution basically statistics and when you're doing some long-running task it tells you basically what is happening and i'm going to show you uh this later on when we do this long running tasks next i'm going to continue with a lot of imports actually and here i'm importing matpot sleep and seabourn for the visualizations that we are probably not going to make in this video next i'm doing pandas in numpy i'm going to import cqdm for the progress bars by torch whitening and we are going to do some min max scaling and then i'm basically importing all stuff related to pytorch and in this particular example we are not going to use many of those but in later videos i'm just going to use some of those and i'm not going to basically scroll up and do the imports again so we do basically a bunch of imports right here and this is executing so i'm going to also set up the macro flip library and the various settings for the color palettes right here and the keyboard styling and then i'm going to import or call tkdm pandas which will allow us to basically monkey patch the apply method for pandas which we are going to use uh in a couple of minutes and finally i'm going to call uh pl dot seed everything so this is from pytorch lightning and this will basically go ahead and see everything including some multi-processing steps that you might want to do and next i'm going to what the data that we are going to use and the data itself is going to be from binance and it's provided by crypto date download.com and here we have historical data for different trading pairs so basically i'm interested in this btc to us dollars minute minute data and this will basically be a lot of data that we are going to use and it's going to contain all those values and i'm going to show you basically what those are i've went and used the the site to download the zip file and then i've extracted the data and from here we have this binance btc usd minute data and it's about 32 megabytes i believe yeah 35 or 36 megabytes so next we are going to use fundus to read the data and this should be from the file and we get out complete here and i'm going to parse the date column and if we look at the data you can see that we have the date the symbol which is going to be exclusively btc usdt and a lot of other values which are going to be all those stuff so we have the opening price the high price the low price the closing price at least for this minute because this is minute by minute data and you can see that the data is probably in reverse order because this is the six minute of the hour fifth fourth etc so what i'm going to do after loading the data is sort the values by date and then i'm going to reset the index of the data frame and right now you can see that the minute by minute data is sorted into the correct order so this is all good and let's check the number of rows that we have we have roughly uh yeah about a quarter of a million rows so this should be well enough data to do basically anything you want and let's start with let's continue with the preprocessing one thing that i occasionally or very often doing is basically calculating some statistics along the current row and the next row or the previous row so for example in this case we might want to have a look at what is the change from the previous closing price for example to the to the current one and one really useful method of pandas is the shift method and if we shift this by one position you're going to see that the let me just show you the close price right here so this is the close price and this is the shifted version by one so you can clearly see that this maps to the zeroth index right here this maps to the second index right here etc etc and we have just uh not a number value at the zeroed index so this is basically just shifting the rows by one and i'm going to use this as a new value or a new column into our data frame and we are going to call this previous calls which should be descriptive enough and if i look at the data right now you can see that we've added this new column at the end of the data frame and the first value is not a number which is going to be a bit of a problem but we are going to handle this right now so we want to basically create a new column which is going to be the closing change price and here i'm going to apply the progress apply function or method which is given to us by the the dkdm pandas library method sorry and in here i'm going to check if the current row is not a number or a none i'm going to use the the numpy method for that we are going to return a zero else we are going to calculate the difference or the delta between the current close price and the previous close price and we are going to do this along the first axis which is going to apply this over uh the rows basically so this will create again another column and as you can see the progress apply is giving us a nice progress bar and here is the number of seconds that we passed during this execution and google co-op is great enough to give us this execution summary i guess so this is a good feature i believe and then we are going to have a look at the new data frame again we have this close change price or the delta and as you can see at the first position or the zeroth index we have zero because it was a non value and after that we are just having the the amount that the closing price has changed i believe that this is in uh probably u.s dollars i believe probably so next i'm going to convert all this into a features data frame that we are going to use for our um next couple of models that we are going to have a look at the next couple of videos and to do this i'm going to create a rows list and in this list i'm going to basically convert of that data frame into features that we're going to use so i'm going to iterate over the data frame and i'm going to do that using tkgm and i'm going to pass in the number of rows that we have so it knows how many iterations we are going to do and i'm going to create a dictionary that is going to contain the data for this row i'm going to get the day of the week so to do that i'm going to access the date day of week property then i'm going to do the same thing for the day of month this is just day next we have the week of year and this can be any features that you want but i've chosen those this is a way to basically encode the date next we have the months months yeah and the opening price the the high price the wall price the close change which is going to be the newly created feature by us and then finally the closing price and i'm going to append this to the rows and i'm going to convert all it into a data frame so here is again tkdm and one interesting thing about the google coop thing is that is jumping all around probably telling us where the current execution point is i don't know but it's doing something all right so after this is complete i'm going to just check the shape of it and then i'm going to have a look at the data frame itself now that the feature creation is complete we have this newly created data frame and it has all the same number of rows that we had before but again we are just doing some feature engineering if you will and in here we have the new numbers extracted from the original data next we are going to make the split between training data and test data and in this case i want to have 90 of the data for training and i'm going to do that by calculating about 90 of the features the number of rows that we have and this will come out to this number next we are going to split the data frame based on this index we are going to create those two data frames and i'm going to use the features data frame and get all of the examples before the train and then i'm going to take all the examples uh sorry the range size after this and i'm going to add one just to make sure that an all data is linked is leaking between the training and test data so these are the numbers that we have for training and testing uh as you can see we have the same number of uh columns or features for each data frame which is good and next we are going to do some transformation of the data and the transformation that we are going to do is basically a min max scaler or scale the data scaling the data allows us to probably get faster convergence when training with stochastic gradient descendant uh and when using probably uh deep neural net models for example lstms or gross or one one confs or maybe you if you are kind of on the cutting edge using transformers from four time series data in all those cases we are basically training with stochastic gradient descent so scaling the features and the labels helps us to get a bit of better results faster convergence and probably is going to fight against the exploding or vanishing gradient problems that you might have so one really easy way to do that is to use this sk1 minmax scaler and in here i'm going to specify the range minus one to one so this will be uh the range that all the features are going to be scaled to and then i want the scaler to fit only the training data and this is very important because we don't want to leak any of the test data that we have thus far into the scaler we want to scale the the data using only the mean values and the standard deviations of the training data that we have so the test data is going to be basically just for testing and we did the same thing here with the split we first got the first n examples and then the later on as the time gets on and on we are reserving this data for testing so this is really important because you don't want it to leak any data from the train from the test data before evaluating it next i'm going to apply the transformation of the scaler and i'm going to call the scaler transform method on the training date frame and i'm going to pass the train df index and the columns yeah this is getting ugly so i'm going to re do this so why we're doing all this basically if you just pass in the scaler transform df let me actually execute this and i'll show you if you pass in the scalar transform train df look at the first row you see that this is no longer a data frame and just to preserve the data frame itself we are going to do the transformation but we are going to basically create a new data frame that is the same data frame that we had before but in here we are going to pass in the index and the columns and of course the transformed values which are going to be this numpy array so i'm going to do the same thing for the test data prime okay so run this and next we let's check what we have into the training date right now so basically the same columns but as you can see the numbers are very very different at least in the price range let's compare this with the previous prices we had rather large values around 10k in u.s dollars and now those values are scaled very differently so this is good it means that it's doing something at least and for the final step of the preprocess we are going to convert those training and test data frames into sequences for that purpose i am going to write a new function which we are going to call create sequences and this function is going to take some input data which is going to be up by uh on this date frame is going to contain the target column the com that we want to predict and the sequence length what we are going to do here then is basically cut the data into multiple sequences and i'm going to start by just writing out the function and then we are going to have a look at an example of how it's going to work so uh please stay with me so the sequences are going to be a list right here and then we are going to check the number of rows into the input data next we are going to iterate over the data and we are going to again use tkdm and i'm going to iterate over the range of data size minus the sequence length so what will this do is iterate over all of the data except for the last examples which are going to be the number of sequences that we have and this is required because otherwise we are not going to have enough uh data at the end to basically go through the sequences so the first thing that i'm going to do is get the sequence and i'm going to use the input data from height index to i plus sequence length so this will give us a sequence with the length of the past parameter that we have and next i'm going to take the next value which is going to be the position of the label and this is going to be i plus sequence length and this is exclusive so we are not getting uh this but we are getting this minus one so this is basically how the indexing in python works so the label position right here is going to be at this position which is inclusive i know this might be a bit of uh hard to get at first but i hope that when we look at the examples this all makes sense next i'm going to take the label value itself and i'm going to take the label position row and from that i'm going to take the target column so this will just give us the row value that we have and next i'm going to append the sequence or a tuple of the sequence and the label finally i'm going to return the sequences and this is pretty much the function that we need uh of course you might go ahead and try something a bit uh more advanced like for example sliding windows or maybe another variation of that this is a really simple implementation in which we are taking every next value and just creating a sequence out of it so let me run this and i have a little example of here right here and i'm going to create a dummy data frame and here i want to create a feature which will have the values yeah one two three four and five and i have uh i'll have a column with the name of label 6 7 8 9 10.

[00:24:43]let's look at the data right here okay so we have this very basic data frame and let's create sequences from this we'll call the function create sequences we are going to pass in the label as target variable and the sequence length is going to be 3.

[00:25:09]let's see so this runs and it's already telling us that it did two iterations so the number of sequences should be two and you might think that one two three two three four and three four five are the possible sequences that you can get but uh that's not exactly correct because for the last sequence you basically don't have a label so let's check the label the sequences themselves and to do that i'm going to first take the sequence at zero position this will give us the sequence and i'm going to print out a row and then i'm going to print out the label at the zeroth position so the first sequence the label itself and this gives us the first three rows from the the data frame and then the label is the next uh the label from the next row so this works as expected and then we have the next example which is going to be the next sequence and for this one we have 2 3 4 2 3 4 7 8 9 and the label is 10. so note that we don't have anymore uh we don't have any more labels so this is why the number of sequences is two all right i hope that all this makes sense and next i'm going to create a sequence length constant i'm going to pass in for example 60 i'm going to create a training sequences using the training data frame and i'm going to pass in the sequence length i'm going to do the same thing for the test sequences and basically related run now that the pre-processing is complete it roughly took about 40 seconds combined here is in the output of that and let's check for example the first sequence what does it have let's first check the label so this is just a number and the sequence is going to be basically this uh i believe that this is a data frame actually yeah it's a date frame which contains all the examples that we want and it should be 59 because it starts from zero so we have a data frame that contains 60 rows and this will be a sequence that is created from the large training data frame this is the shape of it and to basically check how many sequences that we have we have the training sequences and the length of the test sequences right here so this is let's compare this to the to the training data frame so as you can see let me just copy this this is the number of examples that we get into the training date frame and if we do this you can see that this is exactly the so this should be a true if we did everything correctly so this should be exactly the number of the elements that we have into each sequence so everything kind of checks out so this is great and and this is pretty much how you can pre-process your time series data again we had multiple features you can do the same thing basically for a single feature there is no reason that like this shouldn't work you might need to do some fiddling but it should be kinda alright in the next video i'm going to show you how you can convert those sequences into python data sets and then we are going to train an ostm or some other model to basically predict the bitcoin price into the future so we can get rich all of us thanks for watching guys i really would love to for you to do to give me a like maybe subscribe to the channel as well and drop a comment below if you have any questions thanks for watching bye

Up Next

How Outliers Impact Machine Learning Models | CPSC 330

@mgelbart

461 views•2020-11-30

BitTorrent Protocol Explained: Piece Selection & Peer Choking

@StevenGordonAU

481 views•2013-02-22

Multivariate Time Series Forecasting with LSTM in PyTorch | Tutorial

@venelin_valkov

45.7K views•2021-03-30

Enigma Machine Mechanics: WWII Encryption Explained

@JaredOwen

13.2M views•2021-12-11

Related Study Plans & Knowledge Roadmaps

Structured learning paths in Computer Science

View All Computer Science Topics

computer-science

Building BitTorrent: Go, P2P & Concurrency

01Go Fundamentals & Concurrency

02Networking Foundations & P2P Architecture

03Bencode Parsing & Torrent Metadata

+ 3 more learning modules

Explore Roadmap

computer-science

Actor Models (C++ Concurrency & Lock-Free)

01C++ Multi-Threading Foundations

02Thread Pools and Task Scheduling

03Atomics and Lock-Free Queue Design

+ 3 more learning modules

Explore Roadmap

computer-science

Coding HM Type Inference: Algorithm W & OCaml

01Introduction to Functional Programming & OCaml

02Abstract Syntax Trees & Language Interpreters

03Type Systems & Lambda Calculus

+ 2 more learning modules

Explore Roadmap

computer-science

Building OAuth 2.0: Node.js OIDC Server

01Web & Security Foundations

02Authentication vs. Authorization & JWTs

03The OAuth 2.0 Protocol & Grant Types

+ 3 more learning modules

Explore Roadmap