Robotic Arm Dos

One year later…

Last year was my first encounter with the robotic arm. I managed to understand what it is and how to set it up. Sounds superficial? Indeed!.

This spring semester I decided to face the robotic arm again, with a bit more patience and a bit more courage. I felt confident that at least I can learn a bit more on the software side.

Goals:

The GRAND goal was to stack jenga blocks in a closed-loop visual control rather than a scripted coordinate.

To achieve this I give myself some Sub Goals:

1. Using the Intel RealSense D435 to build a clean data collection setup.
   • Meaning that I can create a repeatable camera setup and capture images of Jenga blocks under different lighting, angles, backgrounds, and stack arrangement
2. Using classical OpenCv to detect jenga blocks in the image.
   • edge detection
   • contour detection
   • rectangle fitting
   • oriental estimation
3. Estimate block position and orientation in real space
4. Execute reliable pick actions
5. Execute reliable place actions
6. Add corrections and feedbacks

Outreach:

Cold email is sometimes very hot:

Resources

• UFACTORY Documentation — Official UFACTORY docs for manuals, APIs, support articles, and release notes.
• xArm Python SDK — Official Python SDK for controlling xArm robots.
• UFACTORY Vision — UFACTORY’s vision-related repository, useful for camera integration and vision-based robotic workflows.
• GGCNN Kinova Grasping — Reference project for vision-guided robotic grasping using GG-CNN.

OpenCV resources

• OpenCV-Python Tutorials: Image Processing in OpenCV — Main OpenCV image-processing tutorial hub.

• Camera Calibration — Calibrate the camera and correct distortion.

• Camera Calibration and 3D Reconstruction (calib3d) — Geometry reference for calibration and projection.

• Image Thresholding — Segment blocks from the background.

• Morphological Transformations — Clean up masks after thresholding.

• Contours: Getting Started — Find object outlines in binary images.

• Contour Features — Measure contours and get rotated rectangles.

• Perspective-n-Point (solvePnP) — Estimate object pose from image points.

• Template Matching — Simple baseline for locating known visual patterns.

• Hough Line Transform — Detect long straight edges and estimate orientation.

• LearnOpenCV: Stereo Camera Depth Estimation With OpenCV — Helpful later if you add stereo/depth.