What is a Dense Tensor?

Dense tensors store values in a contiguous sequential block of memory where all values are represented. Tensors or multi-dimensional arrays are used in a diverse set of multi-dimensional data analysis applications. There are a number of software products that can perform tensor computations, such as the MATLAB suite that has even been enhanced by various open source third party toolboxes. MATLAB alone is capable of supporting a variety of element-wise and binary dense tensor operations A dense layer is a fully connected layer, as each and every neuron gets an input from all the neurons in the previous layer, thus being densely connected. This means that every Neuron in a Dense layer will be fully connected to every Neuron in the prior layer. Dense is usually used towards the end of a network, and sometimes multiple times. Trying to build a layered infrastructure for high-performance dense tensor applications, one of the most used libraries is dten, which is known for storing and manipulating dense tensors. The library focuses on storing dense tensors in canonical storage formats and converting between storage formats in parallel. In addition, it supports tensor matricization in different ways. The library is general-purpose and provides a high degree of flexibility. We may regard a tensor as the multidimensional generalization of a matrix. Mathematically, matricization is merely a conceptual (or logical) restructuring of the tensor.

Additional resources on dense tensors

• TensorFlow™ on Databricks

Frequently asked questions

What is a dense tensor?

A dense tensor is a multi-dimensional array in which most elements are nonzero and every value is stored explicitly in a contiguous sequential block of memory. This differs from representations that only track nonzero entries, since a dense tensor keeps all values, regardless of position, readily accessible in memory. Tensors are used across a diverse set of multi-dimensional data analysis applications and can be thought of as the multidimensional generalization of a matrix.

What makes a tensor "dense" rather than storing only nonzero values?

A tensor is considered dense because it stores every value explicitly rather than omitting or compressing entries. All values sit together in a contiguous sequential block of memory, so most elements in the array are nonzero and represented directly rather than inferred. This explicit storage also makes tensor matricization straightforward, since matricization is simply a conceptual or logical restructuring of the tensor's existing values.

What is a dense layer in a neural network, and how does it relate to dense tensors?

A dense layer is a fully connected layer in a neural network, meaning every neuron in that layer receives input from every neuron in the previous layer. This dense connectivity produces the intermediate model activations that dense tensors are used to represent. Dense layers are usually placed toward the end of a network and may appear more than once within the same architecture.

What software tools can perform dense tensor computations?

Several software products can perform dense tensor computations, including the MATLAB suite, which has been enhanced by various open source third-party toolboxes. MATLAB alone supports a range of element-wise and binary dense tensor operations. Another example is the dten library, which focuses on storing dense tensors in canonical storage formats, converting between storage formats in parallel, and supporting tensor matricization in different ways; it is described as general-purpose and highly flexible.

Where are dense tensors commonly used in deep learning workflows?

Dense tensors are commonly used in deep learning to represent data such as images, time series, and intermediate model activations. Libraries and frameworks optimize operations on these tensors so that models can train and run efficiently on both CPUs and accelerators. For a closer look at implementing tensor-based deep learning workloads, see TensorFlow™ on Databricks.