Deep Dive into Robotic Manipulation, Mobile Navigation & Jetson Nano Camera Vision

1. Deep Dive into Robotic Manipulation: Installing ROS 2 Foxy in Windows 10

While Ph.D. research focused on wearable rehabilitation exoskeletons and AI therapists, modern robotic manipulation combines 6/7-DoF arms with camera vision and motion planning frameworks like MoveIt. Before building MoveIt 2, we set up a clean native ROS 2 Foxy Fitzroy environment on Windows 10.

call C:\dev\ros2_foxy\local_setup.bat
:: Terminal 1 — C++ Publisher Node
ros2 run demo_nodes_cpp talker

:: Terminal 2 — C++ Subscriber Node
call C:\dev\ros2_foxy\local_setup.bat
ros2 run demo_nodes_cpp listener

2. MoveIt 2 Source Build & Panda Arm Motion Planning in Windows 10

After verifying ROS 2 Foxy, we build MoveIt 2 from source on Windows 10 and run interactive 7-DoF collision-free trajectory planning with the Franka Emika Panda manipulator in RViz2. Dragging the 3D interactive marker at the end-effector computes inverse kinematics (IK) and executes smooth joint-space plans.

call C:\dev\ros2_foxy\local_setup.bat
call C:\dev\moveit2_ws\install\setup.bat
ros2 launch moveit_resources_panda_moveit_config demo.launch.py

3. [ROS 1 Melodic + MoveIt 1 + PR2/Panda] in Ubuntu 18.04 (WSL2 on Windows 10)

Because many legacy industrial manipulation packages and Gazebo plugins target Ubuntu 18.04 + ROS 1 Melodic, running WSL2 (Windows Subsystem for Linux 2) with an X-Server (VcXsrv / XLaunch) gives full Linux compatibility without dual-booting.

# Install Visual Studio Code inside Ubuntu 18.04 WSL2
sudo apt update && sudo apt install code

# Source ROS 1 Melodic & MoveIt 1 Catkin Workspace
source /opt/ros/melodic/setup.bash
source ~/ws_moveit/devel/setup.bash
export DISPLAY=$(cat /etc/resolv.conf | grep nameserver | awk '{print $2}'):0.0

# Launch MoveIt 1 RViz Motion Planning Demo
roslaunch panda_moveit_config demo.launch rviz_tutorial:=true

4. Deep Dive into Mobile Robots: Probabilistic Robotics & ROS Navigation Stack

Connecting Probabilistic Robotics fundamentals (Bayes filters, EKF/PF localization, occupancy grid mapping) to the ROS Navigation Stack (move_base, amcl, global A*/Dijkstra planner, and local DWA/TEB trajectory planner) inside Windows 10 + WSL2 Linux. This foundation directly led to our ICRA 2025 paper on OPPA (Online Planner's Parameter Adaptation).

0. Introduction: Deep Learning & Real-Time Camera Vision on NVIDIA Jetson Nano

Designed and lectured for the 2021 Sogang University SW Summer Camp (funded by Hyundai Mobis), this 8-part bootcamp guides students with basic Python/C++ knowledge through deploying real-time object detection and custom PyTorch-to-TensorRT transfer learning on the NVIDIA Jetson Nano single-board computer (SBC) with a Raspberry Pi CSI Camera Module V2.

• Primary Upstream Reference: NVIDIA's dusty-nv/jetson-inference (Two Days to a Demo / Hello AI World)
• Download Lecture Slide Decks (Google Drive PDFs):

1. Jetson Nano Preparation (1. Jetson Nano 준비)

Step 1 — Flash JetPack 4.5 SD Card: Download the official JetPack 4.5 microSD image from NVIDIA and flash it onto a 32GB/64GB microSD card using Balena Etcher (ignore Windows "Format Disk" popups after flashing).

Step 2 — Direct Boot vs. Headless Mode (PuTTY + VNC): If you do not have a dedicated HDMI monitor, connect the Jetson Nano's Micro-USB port to your Windows PC, open Device Manager → Ports (COM & LPT) to find the USB Serial COM port, and connect via PuTTY (Serial speed 115200) to complete initial Ubuntu setup. Afterwards, connect via SSH to 192.168.55.1 (Port 22) over USB Ethernet and enable the VNC desktop server (vino-server):

cd /usr/lib/systemd/user/graphical-session.target.wants
sudo ln -s ../vino-server.service ./.
gsettings set org.gnome.Vino prompt-enabled false
gsettings set org.gnome.Vino require-encryption false
gsettings set org.gnome.Vino authentication-methods "['vnc']"
gsettings set org.gnome.Vino vnc-password $(echo -n 'thepassword'|base64)
sudo reboot

# After reboot, set virtual desktop resolution via SSH before opening VNC Viewer:
export DISPLAY=:0
xrandr --fb 1000x400

Step 3 — Command-Line Wi-Fi & CSI Camera Hardware Verification: Connect to Wi-Fi via nmcli and verify the Raspberry Pi CSI Camera V2 using nvarguscamerasrc GStreamer pipelines:

# Connect to Wi-Fi from terminal
nmcli dev wifi list
sudo nmcli --ask dev wifi connect <SSID>
sudo apt-get update && sudo apt-get install nano

# Check CSI Camera device node (/dev/video0)
ls -l /dev/video0

# Test live CSI camera feed on desktop (Mode 0: 3280x2464, Mode 1: resized 60fps)
gst-launch-1.0 nvarguscamerasrc sensor_mode=0 ! nvoverlaysink
gst-launch-1.0 nvarguscamerasrc sensor_mode=1 ! 'video/x-raw(memory:NVMM),width=1280, height=720, framerate=60/1, format=NV12' ! nvvidconv flip-method=0 ! 'video/x-raw,width=400, height=300' ! nvvidconv ! nvegltransform ! nveglglessink -e

2. Linux Essentials & Docker Container Setup (2. Linux 및 Docker Container)

Rather than compiling TensorRT, CUDA, and PyTorch from scratch on the SD card, we clone jetson-inference and launch its pre-configured container via docker/run.sh, which automatically mounts the CSI camera and host directories.

git clone --recursive https://github.com/dusty-nv/jetson-inference
cd jetson-inference
docker/run.sh

# Inside the Docker container (# prompt), test live CSI camera display:
cd build/aarch64/bin
./video-viewer --input-width=400 --input-height=300 csi://0

3. Pre-Trained Object Detection & Python API Walkthrough (3. Object Detection)

We run real-time 91-class MS COCO object detection using ssd-mobilenet-v2 accelerated by NVIDIA TensorRT. On first execution, TensorRT optimizes the network graph for the Jetson Nano Maxwell GPU (taking a few minutes) and caches the serialized engine for instant startup on subsequent runs.

cd /jetson-inference/build/aarch64/bin

# 1. Detect objects on static image and save annotated output
./detectnet.py --network=ssd-mobilenet-v2 images/peds_0.jpg images/test/output.jpg

# 2. Real-time object detection on live CSI camera stream
./detectnet.py --network=ssd-mobilenet-v2 --input-width=400 --input-height=300 csi://0

import jetson.inference
import jetson.utils

net = jetson.inference.detectNet("ssd-mobilenet-v2", threshold=0.5)
camera = jetson.utils.videoSource("csi://0")      # or "/dev/video0" for USB webcam
display = jetson.utils.videoOutput("display://0") # or "my_video.mp4"

while display.IsStreaming():
    img = camera.Capture()
    detections = net.Detect(img)
    display.Render(img)
    display.SetStatus("Object Detection | Network {:.0f} FPS".format(net.GetNetworkFPS()))

4-1. Transfer Learning Preparation (4-1. Transfer Learning 준비)

Training PyTorch SSD models directly on Jetson Nano's 4GB shared RAM requires mounting additional Swap Memory (disabling ZRAM) and storing training datasets on an external exFAT USB drive mounted into the Docker container.

# Expand swap file (on host Ubuntu terminal)
git clone https://github.com/JetsonHacksNano/installSwapfile
cd installSwapfile && ./installSwapfile.sh
free -m

# Enable exFAT USB support and launch Docker with USB volume mounted
sudo apt-get install exfat-fuse exfat-utils
cd ~/jetson-inference
docker/run.sh --volume /media/mobis/mobis_usb:/media/mobis/mobis_usb

cd /jetson-inference/python/training/detection/ssd
wget https://nvidia.box.com/shared/static/djf5w54rjvpqocsiztzaandq1m3avr7c.pth -O models/mobilenet-v1-ssd-mp-0_675.pth

# Download 100 images of target class (e.g. "Apple") to mounted USB drive
python3 open_images_downloader.py --max-images=100 --class-names "Apple" --data=/media/mobis/mobis_usb

4-2. Transfer Learning Training, ONNX Export & Live Inference (4-2. 실습 학습)

Once the OpenImages subset is downloaded, we fine-tune the MobileNet-V1 SSD detector in PyTorch, export the best checkpoint to ONNX, and run real-time TensorRT inference on images, videos, and the live CSI camera.

cd /jetson-inference/python/training/detection/ssd

# 1. Train for 5 epochs (or resume from a saved checkpoint with --resume=models/...)
python3 train_ssd.py --data=/media/mobis/mobis_usb --model-dir=models --batch-size=4 --epochs=5

# 2. Convert PyTorch (.pth) checkpoint to ONNX (ssd-mobilenet.onnx)
python3 onnx_export.py --model-dir=models

# 3. Test custom ONNX model on live CSI Camera (TensorRT auto-builds engine on first run)
detectnet.py --model=models/ssd-mobilenet.onnx --labels=models/labels.txt \
  --input-blob=input_0 --output-cvg=scores --output-bbox=boxes \
  --input-width=400 --input-height=300 csi://0

5. NVIDIA DLI Certification & Troubleshooting Notes (5. Certification 관련 주의사항)

NVIDIA's Deep Learning Institute (DLI) offers two official free self-paced certificate courses for Jetson Nano:

• Getting Started with AI on Jetson Nano (DLI+S-RX-02+V2) ↗
• Building Video AI Applications at the Edge on Jetson Nano (DeepStream) ↗

Important Troubleshooting — Reclaiming 5GB from Mismatched Docker Images: Because the DLI course instructions hardcode a JetPack 4.4 Docker tag while your SD card runs JetPack 4.5, running the DLI command downloads a duplicate 5GB image that fails to start. Check and remove the unused Docker image to free SD card space:

# Check available SD card space and list downloaded Docker images
df -h
sudo docker images

# Remove the mismatched IMAGE ID to immediately recover ~5GB
sudo docker rmi -f <IMAGE_ID>
df -h

6. Production Transfer Learning with NVIDIA TLT / TAO (6. Transfer Learning 방법 & 링크)

For production deployment at higher FPS, train on a desktop/cloud GPU using the NVIDIA Transfer Learning Toolkit (TLT / TAO Toolkit) and NGC pre-trained backbones, prune and quantize to FP16/INT8, and deploy the resulting TensorRT engine onto the Jetson Nano.