dhruva.lifecontact

← blog

[Oct 14, 2026 · 9 min read][research]

Leg Spaghetti

A no-gi berimbolo: the bottom player inverts underneath the standing opponent, their legs tangle, and four seconds later they're sitting on the opponent's back
Lachlan Giles, The NOGI Berimbolo (BJJ Fanatics)

This is a berimbolo.

Did you catch that? Watch it again. One second he's on his back under someone standing over him, the next he's upside down, and then somehow he's sitting on the guy's back. Blink and you miss it.

It took me three months to learn this. And I still remember the first time I saw one, because I wasn't even watching the back take. I was watching the legs. Four legs folding into each other, hooking, crossing, threading through.

There's a whole part of the game built around this. Leg entanglements - ashi garami, 50/50, saddle, all the positions where your legs and your partner's legs turn into one big knot. Most people on the mats just call it leg spaghetti.

I've been trying to teach a computer to see it.

Honestly, I'm still figuring out how to say what I'm working on in one line. The closest I've got is visual intelligence for human movement - machines that can watch people move and actually understand it. Kakashi watches your rolls. My paper was about getting vision-language models to describe movement faithfully. And the further I go, the more everything keeps coming back to one stupidly simple question.

Whose limb is that?

Before a coach, a physio or a robot can tell you anything about how you moved, it needs to know which arm is yours and which leg is mine. Get that wrong and everything built on top of it is wrong too. I've watched the big models describe a scramble beautifully and still hand the trapped arm to the wrong person (that was whose arm is it?). So this time I went down a level, to pixels.

The plan was an auto-labeller. Something that draws a mask over each athlete on every frame, so I can train models on top of it later. Standing up, it works. Most ground positions, it works. Then the legs get involved.

Two athletes in a leg entanglement, and the tracker's masks, which give the leg pointing up to the wrong person
That leg pointing at the ceiling? My tracker says it's red's. It's blue's.

In a leg entanglement, one person's foot is usually sitting right between the other person's legs. Bare foot, bare shin, same skin. So if you ask "which body is this foot closest to?", you get the wrong answer every time, because the foot is surrounded by the wrong person. It's inside one body and belongs to the other.

On the hardest frames my tracker got about 72% of points right. On feet and knees, 55%. Basically a coin flip.

My first thought was that the machine needs to understand how bodies are built. So I tried that, twice.

First, 3D bodies. I fit a 3D human to each athlete with SAM 3D Body, rendered both, and let whichever one was in front own the pixel. It was the best fully automatic thing I tried, 77% against 72%. But it got there by filling gaps, not by fixing who owns what. It actually swapped more limbs than before. And every now and then it put a leg in front of a torso it was clearly behind.

Second, body parts. Meta has a model called Sapiens that labels every pixel of a person - thigh, shin, foot. If it knows a foot is a foot, and which shin it's attached to, ownership should just fall out. It found nobody. No feet, no shins, no people. For a while I thought jiu-jitsu footage was just weird for it.

Turns out it has a size limit. On Meta's own demo photo it works beautifully. Shrink that same photo until the person is 512 pixels tall and it sees almost nothing. My athletes are 150 to 400 pixels tall.

0% 20% 40% 60% my athletes: ~150-400 px 53.9% 1024 px 0.1% 512 px 0% 300 px 0% 200 px
Sapiens on Meta's own demo photo, shrunk down. My athletes live in the shaded bit.

Right idea, wrong tools. At least at this resolution.

So I started dotting.

My review pages show two versions of a mask side by side, and I pick the better one. That tells me which one's better, but not where either one is wrong. So I added dots. A red dot means "this pixel is red's", a blue dot means it's blue's. Sometimes I'm pointing at a mistake. Sometimes I'm just tracing a leg through the knot so the truth is written down somewhere. By the end of the day I'd placed 545 of them.

Red and blue dots on a tangle, shown on the raw frame and on two versions of the masks
Same dots, two versions of the masks. The second one fixes the leg but loses red's other one.

The dots ended up doing two jobs.

The first was keeping score. I can check any version of the masks against them, even ones I build later. And that's where the first surprise came from. The mistakes were there before tracking even started. On the frames the tracker starts from, it got 66% of my dots right, and it matched the raw per-frame masks on every single one. The tracker wasn't making mistakes, it was faithfully carrying them forward. Two models agreeing just meant they were wrong the same way.

The second was fixing things. A dot is exactly the kind of hint SAM is built to take - here's a mask, here are a few clicks, fix it. So every dotted frame gets redrawn, and the tracker carries the fix to the frames around it.

Except it didn't. I hid one corrected frame at a time to see if the fixes around it would reach it, and they didn't, not even from 1.5 seconds away. Every second or so the tracker restarted from its old automatic starting points, and those brought all the mistakes straight back. Once I cleared those out for a few seconds around each correction, it worked. The hidden frames went from 74% to 82%.

Then I reviewed it blind. 21 to 0.

Three tangles, each shown raw, with the old masks and with the corrected masks
Not a single dot on any of these. The fixes came from frames up to 3.6 seconds away.

My note from that review, word for word:

A is near perfect! I'm astounded, this is insane!

All of that from two or three dotted frames per clip.

Now, I do want to be honest about what this is, because it took me a minute to get it straight myself. Nothing learned anything. No model was trained, no weights changed. It's the same trick as clicking on something in SAM's video demo and watching it follow along. Which also means it doesn't carry over. A clip I haven't dotted still gets the old, wrong masks. Getting the machine to do this on its own is a whole different problem, and I haven't cracked it.

The corrections did something I didn't expect, too. They mostly stopped giving limbs to the wrong person, but a lot of the time they didn't give them to anyone. On dots no method had ever seen, swaps dropped from 88 to 33, and holes went up from 43 to 74.

I'm honestly fine with that. A hole can be marked "not sure" later. A swap teaches the next model the wrong thing.

So who gets the hole? I tried two ways.

By outline: give the pixel to whoever's body it sits inside. Right 40% of the time, worse than a coin flip. Which makes sense if you think about it - a trapped foot sits inside the other person's outline. Same problem as before.

By bones: draw each athlete's skeleton (hip to knee to ankle) as thick lines, and give the pixel to whoever's bone runs through it. Right 62% of the time.

A trapped leg in grey leggings, left unowned, then filled by outline (wrong) and by bones (right)
The leg in grey leggings is blue's. Outline gives it to red. Bones give it back.
baseline 24 53 24 pose-guided SAM 46 55 corrections 40 17 44 + fill by outline 60 41 + fill by bones 65 30 0 20 40 60 80 100 dots (n = 101)
rightswappednobody's
101 dots none of these methods ever saw.

Which is pretty much what I'd been saying the whole time, just with a number on it now. A foot belongs to whoever's leg leads to it.

But 62% isn't 90%. This isn't done, not even close.

The skeletons are guessed from the masks, so a bad mask bends the skeleton. The skeleton model stops at the ankle - no feet at all, so I faked them. When your shin crosses my foot, two bones claim the same pixel, and only depth can settle that. Every clip still needs me to sit there and dot it. And the really tangled stuff, the actual spaghetti, still comes out with holes in it, or worse.

When I dot a trapped foot myself, I think I'm using three things. What it looks like, what it's attached to, and where it was a second ago. Skin kills the first one. Bones are a rough version of the second. Nothing in the system uses the third yet, and I'm pretty sure that's the one I lean on most when I'm scrubbing back and forth through a roll.

So here's what I want to try next:

  • Bodies that fit together. Two people can't be in the same place at the same time. A 3D fit that knows that should stop putting legs through torsos.
  • Better 3D. The 3D bodies were my best automatic result, just for the wrong reasons. When two bones fight over the same pixel, depth is the tiebreaker, so I think they deserve a second shot.
  • Human primitives. Break the body into simple parts - thigh, shin, foot - and work out how they connect, instead of treating each person as one big blob.
  • Real foot points. A skeleton that actually has feet, not the ones I made up.
  • Track limbs, not just people. A foot moves with its own shin, not with the leg it's stuck under. Follow pixels through time, not just shapes.
  • Let it be unsure. If it doesn't know whose foot that is, it should say so and wait. Then when the knot comes apart a few seconds later and it's obvious, go back and fill in the frames it skipped. That's literally what I do when I rewind.
  • Learn from the dots. Fine-tune on everything I've dotted so far, so the next clip starts out better than this one did.

Three months to learn the berimbolo. Probably longer to teach machines to see it.

The numbers, for the curious
  • Setup: 11 leg-entanglement clips, about 21 s each, 720p. 545 dots on 62 frames. Where my notes in a blind review disagree with the key I pressed, the notes win.
  • Before corrections, on 362 dots: 72% right. Feet 55%, knees 55%, hips 70%, hands 78%. On the tracker's own starting frames: 66%, identical dot for dot to the raw per-frame masks.
  • 3D bodies: 77% right, with 69 swaps against the tracker's 54. Pose-guided masks (BBoxMaskPose): 59-63%, and they lost a blind review 6-0. Sapiens: no change, because it found no people.
  • Hidden-frame test: with the old starting points left in, 73% → 73% (22 frames). With them cleared for ±3.6 s, 74% → 82% (8 frames, 1.5-3 s from a correction).
  • Blind reviews: frames right next to a correction, 12-5 for the corrected version (1 the same). Frames where the two differ most, 21-0.
  • Filling holes: by outline, 37 of 92 right (40%). By bones, 44 of 71 (62%).
  • Sapiens on its own demo photo: 54% of pixels human at full size, 0.1% when the person is 512 px tall, 0% at 300 px.
Papers, for those who can read

And a few terms, if you need them:

  • Mask: which pixels in a frame belong to which person. Red's, blue's, or nobody's.
  • Swap / hole: a swap is a pixel given to the wrong person. A hole is a pixel given to nobody.
  • Tracking: draw masks on a few frames and let the model carry them through the rest of the clip.
  • Skeleton / pose: points on each joint (hip, knee, ankle) joined up with lines.
  • Blind review: two versions side by side in random order, and I pick the better one without knowing which is which.