Data Science
Data Science combines statistical analysis, machine learning, and domain expertise to extract meaningful insights from data. Explore the latest advancements, techniques, and applications in our Data Science blog posts below.
As a rapidly evolving field, Data Science is at the forefront of innovation in technology and business. From predictive modeling to natural language processing, data science techniques are transforming industries and driving new discoveries.
How does Data Science drive innovation and business growth?
Find the related blogs below to explore how Data Science drives innovation and business growth.
Related Blogs
- Time Series Anomaly Detection – Matrix Profile, STUMPY, Deep Learning and Multivariate Graph Methods
by Durgesh Kekare📋 KEY INSIGHTS Time series anomaly detection (TSAD) identifies unusual patterns in temporal data: point anomalies (a single timestep with an unusually high or low value), contextual anomalies (a value that is normal globally but unusual in its local temporal context — a temperature of 20°C is normal in summer but anomalous in winter), and collective anomalies (a subsequence of consecutive timesteps that is anomalous as a pattern even if individual values are normal — a machine vibrating at an unusual frequency for 30 seconds). The distinction matters for algorithm choice: isolation forest and Z-score are effective for point anomalies;… Read more: Time Series Anomaly Detection – Matrix Profile, STUMPY, Deep Learning and Multivariate Graph Methods - Long-Context LLMs – RoPE, YaRN, FlashAttention, Attention Sinks and Lost in the Middle
by Durgesh Kekare📋 KEY INSIGHTS Long-context language models extend the usable context window from the original 2K–4K token range of early Transformers to 128K, 1M, or even 10M tokens in recent models (Gemini 1.5 Pro: 1M tokens; Claude 3.5 Sonnet: 200K; GPT-4o: 128K). This is not merely a quantitative extension — it qualitatively changes what LLMs can do: 1M tokens can hold an entire codebase, a year of meeting transcripts, thousands of research papers, or an entire book series in context simultaneously, enabling analysis, synthesis, and question-answering that would otherwise require complex retrieval systems. The technical challenge: standard multi-head attention scales as… Read more: Long-Context LLMs – RoPE, YaRN, FlashAttention, Attention Sinks and Lost in the Middle - Data Science for Real Estate – Automated Valuation Models, Hedonic Regression and Market Forecasting
by Durgesh Kekare📋 KEY INSIGHTS Automated valuation models (AVMs) predict residential and commercial property values from property characteristics and market data, replacing or augmenting manual appraisals in mortgage underwriting, property tax assessment, real estate search platforms (Zillow Zestimate, Redfin Estimate), and investment analysis. The core ML problem is hedonic regression: estimate property value as a function of its characteristics (size, bedrooms, bathrooms, age, lot size, condition), location (neighbourhood, school district, proximity to amenities), and market conditions (comparable recent sales, interest rate environment). The hedonic regression framework assumes that a property’s value is the sum of the implicit prices of its individual attributes… Read more: Data Science for Real Estate – Automated Valuation Models, Hedonic Regression and Market Forecasting - Mechanistic Interpretability – Sparse Autoencoders, Circuits, Superposition and Feature Discovery
by Durgesh Kekare📋 KEY INSIGHTS Mechanistic interpretability (MI) is the subfield of AI safety and interpretability that attempts to reverse-engineer the algorithms implemented by neural network weights — to understand not just what a model predicts, but which internal circuits (compositions of attention heads and MLP neurons) compute those predictions, and what concepts each neuron or feature direction represents. Unlike post-hoc attribution methods (SHAP, LIME) that explain individual predictions in terms of input features, MI seeks a complete computational account of the model: the specific sequence of linear and nonlinear operations that transform raw inputs into outputs, expressed in human-interpretable terms. Superposition… Read more: Mechanistic Interpretability – Sparse Autoencoders, Circuits, Superposition and Feature Discovery - Data Science for Energy – Load Forecasting, Renewable Power, Demand Response and Grid Optimisation
by Durgesh Kekare📋 KEY INSIGHTS Data science for energy focuses on five high-value problem areas: short-term load forecasting (predicting electricity demand 1-48 hours ahead for grid dispatch optimisation), renewable energy forecasting (predicting solar irradiance and wind power output for grid balancing), grid stability and fault detection (identifying equipment failures, anomalies, and stability risks in real time), demand response optimisation (incentivising consumers to shift load away from peak hours), and carbon tracking (monitoring and attributing emissions from complex energy systems). Each problem has distinct data characteristics, modelling requirements, and operational constraints. Short-term load forecasting (STLF) predicts electricity demand at 15-minute to 1-hour resolution… Read more: Data Science for Energy – Load Forecasting, Renewable Power, Demand Response and Grid Optimisation - Causal Representation Learning – Identifiability, Disentanglement, IRM and OOD Generalisation
by Durgesh Kekare📋 KEY INSIGHTS Causal representation learning (CRL) studies how to learn representations whose dimensions correspond to causally meaningful, independent factors of variation in the data — representations that are both disentangled (each latent dimension corresponds to one semantic factor) and causally structured (the latent factors have a known or learnable causal graph relating them). The motivation: standard representation learning learns features optimised for predictive performance on the training distribution but that may bundle together causally distinct factors. A representation entangling lighting and shape in a vision model will fail when lighting shifts (domain shift) but shape stays constant, because the… Read more: Causal Representation Learning – Identifiability, Disentanglement, IRM and OOD Generalisation - Multi-Task Learning – Hard vs Soft Parameter Sharing, Gradient Surgery and Loss Weighting
by Durgesh Kekare📋 KEY INSIGHTS Multi-task learning (MTL) trains a single neural network to perform multiple tasks simultaneously, sharing representations across tasks to improve generalisation, reduce training time, and lower inference cost compared to training separate models per task. The key inductive bias: tasks that are semantically related share useful intermediate representations, and sharing a feature extractor across tasks acts as a regulariser — each task provides auxiliary training signal that prevents the shared layers from overfitting to any single task’s training distribution. MTL consistently outperforms single-task learning when tasks are related and training data per task is limited. Hard parameter sharing… Read more: Multi-Task Learning – Hard vs Soft Parameter Sharing, Gradient Surgery and Loss Weighting - Continual Learning – Catastrophic Forgetting, EWC, Progressive Networks and Replay Methods
by Durgesh Kekare📋 KEY INSIGHTS Continual learning (CL) — also called lifelong learning or incremental learning — studies how neural networks can learn a sequence of tasks T_1, T_2, …, T_n without forgetting previously learned knowledge when trained on a new task. The central challenge is catastrophic forgetting (McCloskey & Cohen, 1989): the standard gradient descent update for task T_k overwrites the weight configurations that encoded task T_{k-1}, because the gradient step is computed only on T_k data with no memory of or regularisation toward the previous task. Catastrophic forgetting is not a theoretical curiosity — it is a fundamental obstacle to… Read more: Continual Learning – Catastrophic Forgetting, EWC, Progressive Networks and Replay Methods - Tabular Foundation Models – TabPFN, In-Context Learning and Bayesian Inference for Structured Data
by Durgesh Kekare📋 KEY INSIGHTS Tabular foundation models (TFMs) pre-train on large collections of tabular datasets and perform in-context learning at inference time — classifying or regressing on a new dataset by treating the entire training set as context, without any gradient updates. This is a radical departure from the standard tabular ML pipeline (feature engineering, model selection, hyperparameter tuning) and represents the application of the foundation model paradigm to structured data. The key insight: a model trained on enough diverse tabular datasets learns meta-features about tabular classification problems — the relationship between feature patterns and labels — enabling it to generalise… Read more: Tabular Foundation Models – TabPFN, In-Context Learning and Bayesian Inference for Structured Data - Reward Modelling and RLHF Deep Dive – DPO, Constitutional AI and Preference Learning
by Durgesh Kekare📋 KEY INSIGHTS Reward modelling is the process of learning a scalar reward function from human preference data, used in Reinforcement Learning from Human Feedback (RLHF) to align language models with human values. The pipeline: (1) sample pairs of model outputs (y_1, y_2) for the same prompt x; (2) ask human raters which output they prefer; (3) train a reward model R_φ(x, y) to predict human preferences using the Bradley-Terry model: P(y_1 > y_2 | x) = σ(R_φ(x, y_1) − R_φ(x, y_2)), where σ is the sigmoid function; (4) use RL (typically PPO) to fine-tune the LLM to maximise expected… Read more: Reward Modelling and RLHF Deep Dive – DPO, Constitutional AI and Preference Learning - Data Science for Manufacturing – Predictive Maintenance, Quality Control and Digital Twins
by Durgesh Kekare📋 KEY INSIGHTS Data science for manufacturing focuses on four high-value problem areas: predictive maintenance (predicting equipment failures before they cause unplanned downtime), quality control (detecting defective products and identifying root causes of quality deviations), process optimisation (finding optimal process parameter settings to maximise yield, throughput, or energy efficiency), and demand-driven production planning (aligning production schedules with demand forecasts and supply constraints). Each problem uses a different combination of sensor data, historical records, and domain knowledge about physical processes. Predictive maintenance (PdM) uses sensor data from industrial equipment (vibration, temperature, pressure, current, acoustic emission) to predict when a machine will… Read more: Data Science for Manufacturing – Predictive Maintenance, Quality Control and Digital Twins - Geometric Deep Learning – E(3)-Equivariant GNNs, PointNet and SE(3)-Transformers
by Durgesh Kekare📋 KEY INSIGHTS Geometric deep learning (GDL) is the unifying framework that extends deep learning to data with non-Euclidean structure — graphs, point clouds, meshes, manifolds, and 3D geometric objects — by building neural network architectures that respect the symmetries of the underlying data domain. The core principle: a learning algorithm should be equivariant to the symmetries of its input domain. For images, translational equivariance gives CNNs. For sets, permutation equivariance gives DeepSets and self-attention. For molecules and 3D point clouds, E(3)-equivariance (invariance to rotations, reflections, and translations) is essential — a molecule does not change its properties when rotated… Read more: Geometric Deep Learning – E(3)-Equivariant GNNs, PointNet and SE(3)-Transformers - Machine Learning for Drug Discovery – Molecular GNNs, ADMET Prediction and Generative Chemistry
by Durgesh Kekare📋 KEY INSIGHTS Machine learning for drug discovery applies predictive modelling to accelerate the identification, optimisation, and development of drug candidates. The drug discovery pipeline has four stages where ML adds value: target identification (discovering biological targets whose modulation treats a disease), hit identification (finding molecules that bind the target), lead optimisation (improving binding affinity, selectivity, and drug-likeness of hit compounds), and ADMET prediction (predicting absorption, distribution, metabolism, excretion, and toxicity properties before synthesis). ML reduces the time and cost of each stage by enabling virtual screening of billions of compounds without wet-lab synthesis. Molecular property prediction is the core… Read more: Machine Learning for Drug Discovery – Molecular GNNs, ADMET Prediction and Generative Chemistry - Structured State Space Models – S4, Mamba, Selective SSMs and Long-Range Sequence Modelling
by Durgesh Kekare📋 KEY INSIGHTS Structured State Space Models (SSMs) are a family of sequence models that map input sequences to output sequences via a latent state updated by a linear ODE: x'(t) = Ax(t) + Bu(t), y(t) = Cx(t) + Du(t), where A is the state transition matrix, B is the input projection, C is the output projection, and u(t) is the input signal. Discretised to fixed time steps, this ODE becomes the recurrence x_k = Ax_{k-1} + Bu_k, y_k = Cx_k — identical in form to an RNN cell, but with structured constraints on A that enable efficient training via… Read more: Structured State Space Models – S4, Mamba, Selective SSMs and Long-Range Sequence Modelling - Data Science for Retail – Price Elasticity, Demand Forecasting, Markdown and Assortment Optimisation
by Durgesh Kekare📋 KEY INSIGHTS Data science for retail addresses decisions across the retail value chain: pricing (setting and updating prices to maximise revenue or profit), assortment (deciding which products to carry in which stores), inventory (setting safety stock and reorder points), promotions (planning markdowns and trade promotions), and personalisation (product recommendations and targeted marketing). Each problem has a distinct modelling approach, but all share the need for causal inference — naive correlations between prices and sales reflect both demand elasticity (causal) and endogeneity (prices are raised during high-demand periods, confounding the correlation). Price elasticity of demand quantifies how sensitive sales volume… Read more: Data Science for Retail – Price Elasticity, Demand Forecasting, Markdown and Assortment Optimisation - Optimal Transport in Machine Learning – Wasserstein Distance, Sinkhorn and Domain Adaptation
by Durgesh Kekare📋 KEY INSIGHTS Optimal transport (OT) is a mathematical framework for comparing probability distributions by computing the minimum cost required to transport mass from one distribution to another. The Wasserstein distance W_p(P, Q) = (inf_{γ ∈ Γ(P,Q)} ∫ c(x,y)^p dγ(x,y))^{1/p} measures the distance between two distributions P and Q as the minimum expected transportation cost under the optimal coupling γ, where Γ(P,Q) is the set of joint distributions with marginals P and Q. Unlike KL divergence or Jensen-Shannon divergence, Wasserstein distance is a true metric, is defined even when distributions have disjoint supports, and metrises weak convergence — making it… Read more: Optimal Transport in Machine Learning – Wasserstein Distance, Sinkhorn and Domain Adaptation - Foundation Models Beyond LLMs – Time Series, Vision, Protein and Multimodal Foundation Models
by Durgesh Kekare📋 KEY INSIGHTS Foundation models are large-scale models pre-trained on broad, diverse datasets using self-supervised learning, then fine-tuned or zero-shot adapted for downstream tasks. While the term is most associated with LLMs (GPT-4, Claude, Llama), the foundation model paradigm has been successfully extended to time series (Lag-Llama, TimesFM, MOIRAI), images and video (ViT, SAM, Stable Diffusion, Sora), structured tabular data (TabPFN, CTGAN), genomics (Nucleotide Transformer, DNABERT, ESM-2 for proteins), and multimodal inputs combining text, images, audio, and structured data. Time series foundation models pre-train on large collections of diverse time series data to learn universal time series representations transferable to… Read more: Foundation Models Beyond LLMs – Time Series, Vision, Protein and Multimodal Foundation Models - Neural ODEs and Physics-Informed Neural Networks – PINNs, Neural Operators and Scientific ML
by Durgesh Kekare📋 KEY INSIGHTS Neural Ordinary Differential Equations (Neural ODEs, Chen et al., NeurIPS 2018) define a continuous-depth neural network by replacing the discrete layer-to-layer transformation of a residual network with a continuously-parameterised ODE: dh(t)/dt = f(h(t), t; θ), where f is a neural network, h(t) is the hidden state at continuous “depth” t, and the output is obtained by integrating from t₀ to t₁ using a black-box ODE solver. This reformulation has three important consequences: (1) constant memory cost during training regardless of integration steps (via the adjoint method); (2) adaptive computation (the ODE solver uses more steps for harder… Read more: Neural ODEs and Physics-Informed Neural Networks – PINNs, Neural Operators and Scientific ML - Data Science for HR – People Analytics, Attrition Prediction, Pay Equity and Workforce Planning
by Durgesh Kekare📋 KEY INSIGHTS People analytics (HR analytics, workforce analytics) applies data science to human resources decisions: predicting which employees are at risk of leaving (attrition prediction), optimising compensation structures (pay equity analysis, market benchmarking), forecasting workforce demand (headcount planning), identifying high-potential employees, and improving recruiting pipeline efficiency. The field has grown rapidly as HR data infrastructure has matured — HR Information Systems (HRIS), applicant tracking systems (ATS), performance management platforms, and engagement survey tools generate rich longitudinal employee data. Employee attrition prediction is the most widely deployed HR analytics model: given historical HR data for employees who stayed or left,… Read more: Data Science for HR – People Analytics, Attrition Prediction, Pay Equity and Workforce Planning - Probabilistic Forecasting – Quantile Regression, Conformal Prediction, NGBoost and CRPS
by Durgesh Kekare📋 KEY INSIGHTS Probabilistic forecasting produces a full predictive distribution over future values — not a single point estimate — enabling decision-makers to reason about forecast uncertainty, compute confidence intervals, and optimise decisions under risk. The key output formats are: quantile forecasts (the τ-th quantile q_τ such that the true value falls below q_τ with probability τ, e.g. P10/P50/P90), prediction intervals (a range [l, u] containing the true value with probability 1−α), and full predictive distributions (parametric, e.g. Gaussian/Student-t, or non-parametric, e.g. samples from a generative model). The choice of output format depends on the downstream decision: inventory safety stock… Read more: Probabilistic Forecasting – Quantile Regression, Conformal Prediction, NGBoost and CRPS





















