Humans encode contact experience through exploratory touch, store it, and recall it to guide insertion. TaMeSo-bot builds a robotic counterpart to each stage. The illustration is conceptual and does not depict a neural or algorithmic implementation.

You drop a peg into a hole you cannot see, guided only by what the fingers feel. Small exploratory motions tell you where the rim catches, and you recall how a peg of this shape slid, jammed, and finally seated the last time. Cognitive science calls this tactile memory: the encoding, storage, and retrieval of touch-based experience. It keeps the temporal structure of contact rather than isolated events, and it works together with proprioception rather than in isolation.

TaMeSo-bot mirrors the three stages. A soft wrist lets the robot press into contact without triggering a protective stop, so teleoperated demonstrations capture rich signals from the distributed taxels (encoding). MAT³ compresses each sub-trajectory of tactile, force-torque, pose, and action signals into one vector, and each vector goes into a database with the action that accompanied it (storage). During execution the robot encodes its current contact history as a query and replays the action attached to the nearest stored experience (retrieval).

Why this framing matters. Prior insertion systems name their contact phases in advance and hand the robot a state machine to follow. Tactile memory needs no such labels. Similarity in the learned representation decides which past experience applies right now, so fit, align, and insert emerge from the data instead of being written down. The soft wrist absorbs whatever mismatch retrieval leaves behind, which is what lets a memory built from two peg shapes carry over to five it has never touched.

Figure 1: Masked Tactile Trajectory Transformer (MAT³) for distributed taxels. During training, the encoder learns to reconstruct the states and actions within a time window H, while randomly masking input tokens. After training, the tactile trajectory datasets are encoded into a single representative embedding z that captures the spatio-temporal dynamics of the tactile-action sequences.

Softness for safe data collection. TaMeSo-bot pairs a rigid arm with a highly deformable soft wrist and a two-finger gripper carrying a 3 x 3 array of taxels, each measuring a 3D force. The compliance pays off twice: it absorbs contact forces so teleoperated demonstrations can be collected without triggering protective stops, and it keeps the same contact-rich behavior safe at execution time. Those offline demonstrations become the tactile memory, a database the robot queries instead of rolling out a learned policy.

MAT³ encodes what the fingers felt. The keys of that database come from a bidirectional Transformer over sub-trajectories of taxel readings and actions. Taxels and the action are the base tokens, while force-torque, arm pose, and gripper pose are fused in as auxiliary global context; sinusoidal encodings place each taxel on the sensor grid and the action token at its centre. Training is masked-token reconstruction: a random fraction of the input is hidden and the encoder infers it from the surrounding spatial and temporal context, so task-relevant structure emerges without any subtask labels.

Retrieval is the policy. At execution time the current action token is masked, the encoder and average pooling produce a query embedding, and the $k$ nearest stored embeddings are found by L2 distance over an HNSW index, fast enough for real-time control. One neighbour is sampled and its action is executed, so the database itself acts as a non-parametric policy whose behavior stays bounded by the demonstration data.

Demonstrations were collected on two peg shapes only, and the policy is then asked to insert five shapes it has never touched. MAT³ matches the masking-free variant on the seen pegs and pulls clearly ahead on the unseen ones: 85% versus 71% over 200 trials. We also compare against Action Chunking Transformer (ACT), a parametric policy trained on the same 64 demonstrations with tactile-proprioceptive inputs: it reaches 63.8% on seen pegs but falls to 53% on unseen shapes, its dominant failure being downward force applied before the peg is aligned, which triggers a protective stop. The tactile Transformer baseline stays below 20% throughout.

Real-robot evaluation on seen and unseen pegs

Figure 2: Seen and unseen pegs, with dimensions in mm.

Peg-in-hole task success rate under four different starting positions: 40 trials per peg shape, 10 from each starting position. Best result per row in bold.

Retrieval is what keeps the policy inside the demonstration distribution. A parametric policy has no such guarantee, and ACT extrapolates beyond it, executing insertion-phase behavior before the alignment is done.

The gap widens once the setup itself changes. Under starting positions outside the demonstration range, added friction between peg and fixture, and a tilted grasp, MAT³ succeeds on 57.5% of trials, while MAT³ w/o Mask drops to 29.2%, ACT to 14.2%, and the tactile Transformer to 7.5%, evidence that masked training buys robustness rather than just accuracy.

Real-robot evaluation under unseen conditions

Different starting positions, increased friction between the peg and the fixture surface, and a tilted grasping pose.