Matthew N. Bernstein

Matthew N. Bernstein

Machine learning scientist at Stellaromics

Intrinsic dimensionality

6 minute read

Published:

In my formal education, I found that the concept of “intrinsic dimensionality” was never explicitly taught; however, it undergirds so many concepts in linear algebra and the data sciences such as the rank of a matrix and feature selection. In this post I will discuss the difference between the extrinsic dimensionality of a space versus its intrinsic dimensionality.

Introduction

An important concept in linear algebra and the data sciences is the idea of intrinsic dimensionality. I found that in my formal education this concept was never explicitly taught; however, it undergirds so many concepts in linear algebra and data analysis. In this post I will discuss the difference between the extrinsic dimensionality of a space versus its intrinsic dimensionality. These general ideas provides a nice framework for understanding such diverse concepts as the rank of a matrix in linear algebra as well as dimension reduction and feature selection in machine learning.

What exactly is a “space”?

Before jumping into the dimensionality of spaces, let’s first address a very basic question: what is a space? According to Wikipedia, a space is a set of objects with some “structure.” This definition is extremely general, perhaps even trivial, but I think it deserves emphasizing that a space is, first and foremost, a set. When we refer to “three-dimensional space”, we are in essence describing the set of all points in three dimensions. As I’ve said, this may seem trivial at first, but I think it will be an important idea as we move on to describe the concept of “intrinsic dimensionality.”

What is a dimension?

Let’s move on to another basic question: what does it mean for a space to be three-dimensional versus two-dimensional? More generally, what does it mean for a space to be $D$-dimensional? A basic answer to this question is that a $D$-dimensional space is one in which $D$ pieces of information (i.e., characteristics) are required to describe each object in that space. These $D$ pieces of information are called dimensions. For example, in three-dimensional Euclidean space (3D space), we need three pieces of information to describe each point: its value along the x-axis, its value along the y-axis, and its value along the z-axis:

drawing

Intrinsic vs. extrinsic dimensionality

Now that we have some basic ideas down – namely, “space” and “dimensions” – let’s move on to the core of this blog post: intrinsic dimensionality. Before we move on, let me spoil the ending: the intrinsic dimensionality of a space is the number of required pieces of information that we need to describe each object in the space. This may differ from the number of pieces of information that we are using, which we call the extrinsic dimensionality of the space.

Let’s make this concrete with an example. Let’s say we’re in some situtation we’re we are dealing with objects in some extrinsically $D$-dimensional space. Thus, we are dealing with $D$ pieces of information for describing objects in that space. One may ask a simple question: do we really need $D$ pieces of information to describe each object? Or can we get by with fewer?

Take the following example: we want to describe all points on a flat piece of paper in 3D space. That is, all of the points that we care about will lie only on the sheet of paper.

drawing

Of course, we can use each points x, y, and z coordinates, but do we really need three pieces of information to describe each point on the paper? The answer is no, we really only need two! Intuitively, we can specify each point on the paper using two coordinates: its distance from the left edge of the paper and its distance from the top edge of the paper:

drawing

Of course, representing each point on the paper using these new coordinates requires keeping track of the position of the left edge and top edge in 3D space; however, once we have this information handy, then every point on the paper can be represented using only these two coordinates.

The intrinsic dimensionality of a space is the number of required pieces of information for representing each object. In the piece of paper example, only two coordinates are needed to describe each point on the paper, and thus, it can be said that the “space” of the paper (i.e., the set of all points that lie on the paper) is intrinsically only two-dimensional rather than three-dimensional. Notably, the intrinsic dimensionality of a space may be different than its explicit dimensionality. That is, even though we may be representing each point on the paper using their original three coordinates in 3D space, we could instead only use two.

When will the intrinsic dimensionality of a space be smaller than its extrinsic dimensionality? Intuitively, this will happen when the space that we care about can be formed by taking a subset of the full extrinsic space. In the piece of paper example of above, we only care about the subset of points in 3D space that lie on the piece of paper. By taking a subset, we are in essence coming up with a new, smaller space than the full 3D space. Such spaces are called subspaces.

Another example

Here’s another example of a piece of paper embedded in 3D space, but this time, the paper is rolled up in a “Swiss roll” shape:

drawing

Again, one may say that the set of points that lie on the Swiss roll forms an intrinsically two dimensional space rather than a 3D space. The reason is that we can represent each point using two pieces of information: its distance along the width of the roll and its distance around the roll:

drawing

This is very high level and we haven’t described a rigorous system of coordinates for actually describing each point on the roll, but intuitively one can see that if one fixed the geometry of the roll, one could easily describe each point on the roll with only two dimensions.

You May Also Enjoy

Notes on The Art of War by Sun Tzu (Books 1 and 2)

27 minute read

Published:

I am currently reading Sun Tzu’s Art of War and, as cliche as it sounds, am finding much wisdom in it. I have been taking notes during my reading and I thought I’d share them in this post. Here I cover Books 1 and 2.

Denoising diffusion probabilistic models (Part 2: Theoretical justification)

11 minute read

Published:

In Part 1 of this series, we introduced the denoising diffusion probabilistic model for modeling and sampling from complex distributions. We described the diffusion model as a model that can generate new samples by learning how to reverse a diffusion process. In this post, we provide more theoretical justification for the objective function used to fit diffusion models and make connections between the diffusion model and other concepts in statistical inference and probabilistic modeling.

Denoising diffusion probabilistic models (Part 1: Definition and derivation)

56 minute read

Published:

Diffusion models are a family of state-of-the-art probabilistic generative models that have achieved ground breaking results in a number of fields ranging from image generation to protein structure design. In Part 1 of this two-part series, I will walk through the denoising diffusion probabilistic model (DDPM) as presented by Ho, Jain, and Abbeel (2020). Specifically, we will walk through the model definition, the derivation of the objective function, and the training and sampling algorithms. We will conclude by walking through an implementation of a simple diffusion model in PyTorch and apply it to the MNIST dataset of hand-written digits.

Assessing the utility of data visualizations based on dimensionality reduction

14 minute read

Published:

We human beings use our vision as our chief sense for understanding the world, and thus when we are confronted with data, we try to understand that data through visualization. Dimensionality reduction methods, such as PCA, t-SNE, and UMAP, are approaches designed to enable the visualization of high-dimensional data. Unfortunately, because these methods inevitably distort aspects of the data, these methods are receiving new scrutiny. In this post, I propose that dimensionality reduction requires a “probabilistic” framework of interpretation rather than a “deterministic” one wherein conclusions one draws from a dimensionality reduction plot have some probability of not actually being true of the data. I will propose that this does not mean these plots are not useful. Rather, to evaluate their utility, I will argue that empirical user studies of these methods will shed light on whether these methods provide more benefit or more harm in practice.