Skip to content

feat: pose estimation#1100

Merged
chmjkb merged 24 commits intomainfrom
@chmjkb/pose-estimation
May 5, 2026
Merged

feat: pose estimation#1100
chmjkb merged 24 commits intomainfrom
@chmjkb/pose-estimation

Conversation

@chmjkb
Copy link
Copy Markdown
Collaborator

@chmjkb chmjkb commented Apr 24, 2026

Description

Adds a new pose estimation module to react-native-executorch, mirroring the object detection API surface.

  • Native (C++): pose estimation runtime built on VisionModel, taking only modelSource + optional normMean / normStd. numKeypoints is derived from the model's output tensor
    shape — no keypointNames plumbed through the bridge.
  • TS API: PoseEstimationModule (fromModelName / fromCustomModel) and usePoseEstimation hook. Return type is statically tied to keypointMapRuntime maps the native positional array into the named record at
    the boundary, in both forward and the runOnFrame worklet.
  • Built-in models: yolo26n-pose (i will update more model sizes soon)
  • Custom models: supported via fromCustomModel with user-supplied keypointMap; output type follows. Required model contract (input shape, optional normalization, three
    output tensors boxes / scores / keypoints) is documented.
  • Docs: new usePoseEstimation and PoseEstimationModule pages following the object detection structure, plus VisionCamera integration entry.

Introduces a breaking change?

  • Yes
  • No

Type of change

  • Bug fix (change which fixes an issue)
  • New feature (change which adds functionality)
  • Documentation update (improves or adds clarity to existing documentation)
  • Other (chores, tests, code style improvements etc.)

Tested on

  • iOS
  • Android

Testing instructions

  • Run native sanity tests and verify that they pass
  • Run the computer vision example app, both for vision camera and static images.

Screenshots

Related issues

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Additional notes

@chmjkb chmjkb self-assigned this Apr 28, 2026
@chmjkb chmjkb added feature PRs that implement a new feature model Issues related to exporting, improving, fixing ML models labels Apr 28, 2026
@chmjkb chmjkb added this to the v0.9.0 milestone Apr 28, 2026
@chmjkb chmjkb linked an issue Apr 28, 2026 that may be closed by this pull request
@chmjkb chmjkb marked this pull request as ready for review April 28, 2026 12:59
@chmjkb chmjkb requested a review from benITo47 April 28, 2026 12:59
@chmjkb
Copy link
Copy Markdown
Collaborator Author

chmjkb commented Apr 28, 2026

Also, I am aware that there is significant overlap between this code and object detection. I will try to do something about it, but let's merge this first and let's do it in a separate PR.

tests are coming tho

@msluszniak msluszniak self-requested a review April 28, 2026 13:16
Comment thread apps/computer-vision/app/pose_estimation/index.tsx Outdated
@msluszniak
Copy link
Copy Markdown
Member

Please rebase and resolve conflicts. Also, when I merged this one: 6acd575, I forgot to update README.md inside tests directory. But from now, you need to add all files used by a test suite to models_for_test (see attached commit). If you wouldn't mind, I send an update in README.md about this in your PR.

Copy link
Copy Markdown
Member

@msluszniak msluszniak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lack of PoseEstimationTask.tsx blocks testing.

Comment thread apps/computer-vision/app/vision_camera/index.tsx
@chmjkb chmjkb requested a review from msluszniak April 30, 2026 11:12
@chmjkb chmjkb force-pushed the @chmjkb/pose-estimation branch from 09733b7 to b22762e Compare April 30, 2026 11:42
@chmjkb chmjkb force-pushed the @chmjkb/pose-estimation branch from 45ca941 to 1dc017c Compare April 30, 2026 13:17
Comment thread apps/computer-vision/components/vision_camera/tasks/PoseEstimationTask.tsx Outdated
Comment on lines +43 to +44
// TODO: maybe create a ContractNotMet error or something like this, this
// would also need to be applied for other models
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, do we have an issue for that?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'll create one

@msluszniak msluszniak removed this from the v0.9.0 milestone May 4, 2026
Copy link
Copy Markdown
Member

@msluszniak msluszniak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please change names in demo app (see attached screens).

Image Screenshot_20260504_134438_computer-vision

And I also tested vision camera integration and well. I'm not satisfied in 100% with the results. There are plenty of some random lines that go into the corner, etc. Here is a demo of the problems.

Screen_Recording_20260504_134403_computer-vision.mp4

And please add 'letterboxing' to cspell ignore, there is a warning about this one.

@chmjkb
Copy link
Copy Markdown
Collaborator Author

chmjkb commented May 5, 2026

Please change names in demo app (see attached screens).

Image Screenshot_20260504_134438_computer-vision
And I also tested vision camera integration and well. I'm not satisfied in 100% with the results. There are plenty of some random lines that go into the corner, etc. Here is a demo of the problems.

Screen_Recording_20260504_134403_computer-vision.mp4
And please add 'letterboxing' to cspell ignore, there is a warning about this one.

it was only happening for front camera (in portrait orientation), only on Android, pretty nasty to debug but the changes in FrameTransform should fix it.

@msluszniak
Copy link
Copy Markdown
Member

And please add 'letterboxing' to cspell ignore, there is a warning about this one.

This one is still not addressed.

@chmjkb
Copy link
Copy Markdown
Collaborator Author

chmjkb commented May 5, 2026

done cc @msluszniak

@msluszniak
Copy link
Copy Markdown
Member

Ok, android works perfectly fine, now someone needs to test iOS and I think it's ready to ship :)

@benITo47
Copy link
Copy Markdown
Contributor

benITo47 commented May 5, 2026

I ran iOS app. The model downloads and runs correctly. Both on static images and VC feed. GJ.
Approved.
Godspeed

@chmjkb chmjkb merged commit d97781d into main May 5, 2026
5 checks passed
@chmjkb chmjkb deleted the @chmjkb/pose-estimation branch May 5, 2026 12:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature PRs that implement a new feature model Issues related to exporting, improving, fixing ML models

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add pose estimation support

3 participants