How can two interpretations preserve the same mathematical structure?

1. A part-to-whole prediction

We begin after candidate parts and their estimated positions have been supplied. We hold shape, scale, and orientation fixed and allow position to vary. The vertical coordinate increases upward.

Suppose a represented ear position is $(3,5)$ . We describe it with the matrix

M= \begin{pmatrix} 1&0&3\\ 0&1&5\\ 0&0&1 \end{pmatrix}.

The top two entries of the final column give the position. The extra row and column allow translations to be expressed through matrix multiplication.

Suppose the model’s part-whole relationship places the rabbit’s center two units below the ear. We write this relationship as

W= \begin{pmatrix} 1&0&0\\ 0&1&-2\\ 0&0&1 \end{pmatrix}.

Multiplication produces the vote

V=MW= \begin{pmatrix} 1&0&3\\ 0&1&3\\ 0&0&1 \end{pmatrix}.

The estimated rabbit-center position is therefore $(3,3)$ .

The operation does not turn an ear into a rabbit. Under the cognitive interpretation, it transforms the parameters of a representation of a part into the parameters of a prediction about a whole. We specify the part-whole relationship here rather than model how it is learned.

2. The two interpretations

Under the cognitive interpretation, $M$ describes the parameters of a mental representation of the ear’s position. The matrix $W$ characterizes the relationship used to estimate the whole’s position from that representation. The output $V$ describes the parameters of the resulting prediction.

Under the neuronal interpretation, $M$ describes coordinates obtained from an input pattern of population activity. The matrix $W$ specifies the effective transformation of those coordinates, and $V$ describes the corresponding output coordinates. These coordinates need not exhaust the details of the population activity.

The distinction concerns what is described, not how the multiplication is performed. A matrix that describes a mental representation is not thereby that representation. A matrix that describes neuronal activity is not thereby that activity. Sharing a mathematical description does not, by itself, identify the things described.

3. Preserving the transformation

Let $F\colon X\to Y$ be the mathematical transformation from an input space $X$ to an output space $Y$ . For the example above,

F(M)=MW,

with $W$ held fixed.

Now let $c$ be a cognitive input state and $G_C(c)$ the output of the modeled cognitive process. We use $q_{C,\mathrm{in}}$ and $q_{C,\mathrm{out}}$ to assign mathematical coordinates to the cognitive input and output states, respectively. Compatibility requires

q_{C,\mathrm{out}}\bigl(G_C(c)\bigr) = F\bigl(q_{C,\mathrm{in}}(c)\bigr).

On the left, we apply the cognitive process and then describe its output mathematically. On the right, we describe the input mathematically and then apply the formal transformation. The two routes must agree.

For the neuronal interpretation, let $n$ be a neuronal input state and $G_N(n)$ the output of the modeled neuronal process. With corresponding coordinate mappings, we require

q_{N,\mathrm{out}}\bigl(G_N(n)\bigr) = F\bigl(q_{N,\mathrm{in}}(n)\bigr).

These are commutativity conditions: The interpreted and formal routes agree. The coordinate mappings describe states; they are not additional processes connecting the cognitive and neuronal ontologies.

Suppose a cognitive input and a neuronal input have matching mathematical descriptions:

q_{C,\mathrm{in}}(c)=q_{N,\mathrm{in}}(n).

The compatibility conditions then give

\begin{aligned} q_{C,\mathrm{out}}\bigl(G_C(c)\bigr) &=F\bigl(q_{C,\mathrm{in}}(c)\bigr)\\ &=F\bigl(q_{N,\mathrm{in}}(n)\bigr)\\ &=q_{N,\mathrm{out}}\bigl(G_N(n)\bigr). \end{aligned}

Inputs with matching descriptions therefore produce outputs with matching descriptions. The correspondence is preserved through the modeled transformation rather than merely asserted between two isolated states.

The mappings need not be one-to-one. Several neuronal states, for example, may share the same formal coordinates. We therefore require a structure-preserving correspondence, not a one-to-one correspondence between cognitive and neuronal details.

The coordinate mappings must remain fixed across the modeled inputs, rather than being chosen anew after each output is observed. For a model with several stages, a compatibility condition should hold at each modeled stage, not merely between the initial input and final output.

4. A population-level construction

The Neural Engineering Framework distinguishes represented variables from the neuronal activity used to carry them and uses decoding transformations to recover estimates of those variables (Stewart 2012). We use this distinction here in a deliberately linear construction.

First, stack the columns of each $3\times3$ matrix into a nine-component vector:

\mathbf x=\operatorname{vec}(M), \qquad \mathbf y=\operatorname{vec}(V).

The operation $V=MW$ becomes

\mathbf y=K\mathbf x, \qquad K=W^{\mathsf T}\otimes I_3.

Here, $W^{\mathsf T}$ is the transpose of $W$ , $I_3$ is the $3\times3$ identity matrix, and $\otimes$ denotes the Kronecker product. The Kronecker product forms the $9\times9$ block matrix that carries out the original multiplication on the stacked entries. No new operation has been introduced; we have changed its notation.

Let $\mathbf r$ and $\mathbf s$ collect the input and output population-activity variables. Introduce matrices that extract the formal coordinates:

\mathbf x=D_{\mathrm{in}}\mathbf r, \qquad \mathbf y=D_{\mathrm{out}}\mathbf s.

The population vectors can have different lengths and can contain many more than nine components. The decoding matrices specify which aspects of population activity the shared model describes.

Suppose the population transformation is idealized as

\mathbf s=A\mathbf r.

To make the compatibility requirement hold for every input population vector, impose

D_{\mathrm{out}}A=K D_{\mathrm{in}}.

Then

D_{\mathrm{out}}\mathbf s =D_{\mathrm{out}}A\mathbf r =K D_{\mathrm{in}}\mathbf r =K\mathbf x.

Transforming the population activity and then extracting its output coordinates gives the same result as extracting the input coordinates and applying the formal transformation.

An explicit algebraic construction is available whenever we can choose an output encoding matrix $B$ such that

D_{\mathrm{out}}B=I_9.

This means that $B$ embeds a nine-component output vector in a population pattern from which $D_{\mathrm{out}}$ recovers that vector. Setting

A=B K D_{\mathrm{in}}

then satisfies the compatibility condition, because

D_{\mathrm{out}}A =D_{\mathrm{out}}B K D_{\mathrm{in}} =K D_{\mathrm{in}}.

This constructs a compatible linear population model. Whether such a construction respects the physiological constraints of a proposed circuit remains a separate test.

The matrix $A$ is an effective activity-to-activity map, not necessarily a matrix of synaptic weights. It need not equal $W$ or $K$ , and its dimensions may differ from theirs. What links the population transformation to the shared mathematical operation is the compatibility condition.

For a less idealized neuronal model, we can measure the discrepancy between the two routes over a specified range of inputs instead of requiring exact equality.

5. Extension: Combining predictions

The single vote is not yet a detector. We can extend the model by combining predictions from two candidate parts. This follows the general idea of agreement among votes, but the rule below is our simplification, not the expectation-maximization (EM) routing procedure used by Hinton et al. (2018).

Let $a_1,a_2\in\{0,1\}$ indicate whether the two candidate part representations are active. When $a_1=a_2=1$ , let $\mathbf v_1$ and $\mathbf v_2$ be their predicted rabbit-center positions. Suppose an ear and a paw yield

\mathbf v_1= \begin{pmatrix}3\\3\end{pmatrix}, \qquad \mathbf v_2= \begin{pmatrix}3.2\\2.8\end{pmatrix}.

Their average is

\bar{\mathbf v} =\frac{\mathbf v_1+\mathbf v_2}{2} =\begin{pmatrix}3.1\\2.9\end{pmatrix}.

We measure disagreement by the average squared Euclidean distance from that estimate:

E=\frac{1}{2}\left( \left\|\mathbf v_1-\bar{\mathbf v}\right\|^2 + \left\|\mathbf v_2-\bar{\mathbf v}\right\|^2 \right).

For these values, $E=0.02$ . A small value indicates that the two parts support similar whole-object positions.

Let $\varepsilon^2$ be the maximum tolerated disagreement, where $\varepsilon>0$ . Define the detection output as

z= \begin{cases} 1,&\text{if }a_1=a_2=1\text{ and }E\leq\varepsilon^2,\\ 0,&\text{otherwise} \end{cases}.

With $\varepsilon=0.2$ , the example satisfies the criterion because $0.02\leq0.04$ .

Under the cognitive interpretation, the output $z=1$ marks an occurrence of RABBIT; under the neuronal interpretation, it marks a specified output population state. The position estimate $\bar{\mathbf v}$ and the detection output $z$ have different roles: One specifies where the candidate whole is estimated to be; the other specifies whether the detection criterion is met.

Agreement is not the same as correctness. Two predictions can agree and still be wrong. Nor does satisfying this criterion by itself establish what makes RABBIT mean rabbit.

Each of the averaging, disagreement, and thresholding operations needs its own interpreted processes and compatibility conditions. The linear population construction above covers the vote-producing stage, not the entire detector.

6. Variations of the comparison object

Change the part-whole relationship. Replace $W$ with $W+\Delta W$ . The change in the vote is

\Delta V=M\Delta W.

In the numerical example, moving the assumed rabbit center from two units below the ear to three units below it changes the predicted center from $(3,3)$ to $(3,2)$ .

Cognitively, the system uses a different part-whole relationship. Neuronally, the effective transformation of the population coordinates changes correspondingly. We can ask whether RABBIT’s content remains stable through this change in how the system estimates a rabbit’s position, rather than treating every change in $W$ as a change in meaning.

Translate the represented scene. Let $T$ apply the same translation to each input pose. Then

F(TM)=(TM)W=T(MW)=TF(M).

The vote shifts in the same way as the input pose. This is an equivariance property of the vote-producing operation: A transformation of the input produces the corresponding transformation of the output.

For the two-part extension, the predicted positions and their average all shift by the same amount. Their differences remain unchanged, so $E$ remains unchanged. With the activation indicators and the threshold held fixed, $z$ is therefore unchanged. The predicted position is translation-equivariant, while the detection decision is translation-invariant.

These are properties of the specified model. Whether a change in an actual scene produces the required changes in the system’s internal coordinates is a test for a proposed interpretation.

Change the population transformation while preserving the shared operation. Suppose $A$ satisfies the compatibility condition. Choose a variation $\Delta A$ such that

D_{\mathrm{out}}\Delta A=0.

Then

D_{\mathrm{out}}(A+\Delta A) =D_{\mathrm{out}}A =K D_{\mathrm{in}}.

The population transformation changes, but its effect on the modeled output coordinates does not. This gives us a controlled way to compare physical differences that are invisible at the resolution of the shared model. Permissible variations must still respect the constraints of the chosen physical interpretation.

We could also replace the neuronal interpretation with one involving silicon-based circuits. The physical states, processes, and coordinate mappings could differ while preserving the same formal transformation $F$ . This would make the proposed comparison with multiple realizability mathematically explicit.

7. What the construction establishes

The equations specify how two interpretations can preserve one mathematical transformation. They do not select the interpretations for us. A cognitive interpretation must specify the representational roles attributed to its states and processes. A neuronal interpretation must specify the physical variables and transformations it describes. Matching a sequence of numbers is not a substitute for either task.

We keep psychological predicates within the cognitive interpretation so that their application to neuronal processes is not silently assumed. The neuronal interpretation describes physical states and transformations; the mathematical model makes the structural correspondence between the two interpretations explicit.

The example begins with a specified cognitive interpretation; it does not derive semantic content from matrix multiplication or agreement among votes. What the mathematics supplies is a way to examine the arrangement. We can change a relation or operation, calculate the consequences, and ask whether both interpretations still fit. The comparison object makes those questions explicit without settling them in advance.