# LOTA — LiDAR Over The Air

> LOTA is an iOS app that turns iPhones into professional spatial capture and streaming tools. Stream real-time depth, color, point cloud, motion capture, IMU, audio analysis, and live transcription data over NDI, TCP/UDP, OSC, and PLY. Export COLMAP and Nerfstudio datasets for Gaussian Splat and NeRF training, capture PBR material sets from flat surfaces, and inspect saved point clouds in an on-device 3D viewer. Built for creative professionals using TouchDesigner, OBS, vMix, Resolume, Max/MSP, Ableton, Unreal, Unity, Blender, and other live visual / 3D tools.

- Requires iOS 26.2+. LiDAR (iPhone 12 Pro or later) needed for Depth, Point Cloud, Blob Track, Material Capture, and LiDAR-based 3D-scene recording formats. Color, Mono, Neural Depth (Depth Anything V2 on the Apple Neural Engine), Transcription, Motion, and Audio modes work on every iPhone. **LOTA is officially iPhone-only** — iPad Pro with LiDAR has run during beta testing but is not an officially supported target
- Nine capture modes on the Camera / Streaming page: Color, Mono, Depth (9 LiDAR colormaps), Neural Depth, Point Cloud, Blob Track, Transcription, Motion, Audio
- ARKit Tracking page: body (91 joints, 18 key joints over OSC), face (52 blend shapes), hand tracking (21 landmarks per hand, up to 2 hands)
- Gaussian Capture page: COLMAP, Nerfstudio, Nerfstudio + Depth, Point Cloud PLY, Material (PBR), IMU Trace, Audio Trace, and (new in v1.2.6) PLY Viewer
- GPU-accelerated via Metal compute shaders, adaptive thermal throttle, on-device processing, no analytics
- Fully accessible: VoiceOver, Voice Control, Dynamic Type, Differentiate Without Color, Increase Contrast, Reduce Motion

---

## Getting Started

LOTA turns your iPhone into a professional spatial capture and streaming tool. Three swipeable pages cover live camera modes and streaming (center), ARKit motion capture (swipe left), and dataset / material / trace recording plus a built-in PLY viewer (swipe right).

### Requirements

- iOS 26.2 or later
- LiDAR-equipped iPhone (12 Pro or later) for Depth, Point Cloud, Blob Track, Material Capture, and LiDAR-based 3D-scene recording formats
- Color, Mono, Neural Depth, Transcription, Motion, and Audio modes work on every iPhone — no LiDAR needed
- Wi-Fi network (for streaming features). 5 GHz strongly recommended

### Availability

LOTA is currently in public TestFlight beta. The full App Store release is coming soon. Join the beta at https://testflight.apple.com/join/jFNkCjNF.

### First Launch

The welcome screen lists the seven iOS permissions LOTA needs — **Camera, Microphone, Speech Recognition, Local Network, Location** (for compass heading), **Motion & Fitness**, and **Photos** (add-only, for saving local recordings to your camera roll) — and a one-paragraph privacy reassurance. Tap **Continue** and iOS surfaces the seven permission dialogs in sequence. Granting upfront avoids dialog cascades mid-shoot. The Photos prompt is the narrower add-only variant, so LOTA can write videos but never read existing photos. Denying any one is fine — the app still works for non-permission features and the in-mode fallback UI covers denials.

In the background, LOTA prefetches the on-device speech recognition model for your locale so Transcription mode works without a download wait the first time you use it.

After permissions, a 10-step guided tour walks you through the three pages. Replay it anytime via **Transmission Settings → Help → Replay Tutorial**.

### Quick Start

1. Install LOTA via TestFlight and grant the requested permissions on first launch.
2. Tap the mode dropdown at the top of the Camera / Streaming page to choose a capture mode.
3. Tap the shutter at the bottom to **Stream** (over the network) or **Record** (locally to Photos). For 3D dataset capture, swipe right to the Gaussian Capture page, pick an export format, and record.

---

## Navigation

The app has three swipeable pages:

| Swipe | Page | Purpose |
|-------|------|---------|
| Swipe left | ARKit Tracking | Body, face, and hand motion capture |
| Center (default) | Camera / Streaming | Live camera modes with network streaming + local recording |
| Swipe right | Gaussian Capture | Record datasets for 3D reconstruction, capture PBR materials, export IMU / Audio Trace files, inspect saved point clouds |

---

## Capture Modes (Center Page)

Tap the **mode picker** at the top of the screen (a glass capsule showing the active mode's icon plus a chevron). A glass panel drops down with all nine modes laid out in a grid. The active mode is highlighted yellow; LiDAR-only modes dim and disable on non-LiDAR devices.

### Color

Live RGB camera feed at 60 FPS. Works on every iPhone — no LiDAR required.

### Mono

High-contrast grayscale feed optimized for low-light environments. Works on every iPhone.

### Depth

LiDAR depth visualization with 9 selectable colormaps (Black & White, Blue Red, Deep Sea, Incandescent, Heated Metal, and others). Tap the colormap picker to cycle through them in real time. Requires LiDAR.

### Neural Depth

AI-estimated depth from the regular camera feed. Runs **Depth Anything V2 Small** (ByteDance Research, Core ML packaging by Apple, Apache 2.0, ~24.8 M parameters) on the Apple Neural Engine at ~30 ms per frame. Works on every iPhone — no LiDAR required. LiDAR-equipped phones can still pick Neural Depth as a comparison source. A loading overlay appears the first time the model warms up.

### Point Cloud

Streams the LiDAR point cloud over PLY. Every pixel of the LiDAR depth map is unprojected into 3D in parallel via Metal compute shaders and sent over the network; the on-device view shows the live RGB camera, so the geometry is visualized receiver-side. Configurable Max Depth (1–10 m) and Compute Quality (Full / Balanced / Efficient, where the rate strides on unique LiDAR scans). Requires LiDAR.

### Blob Track

TouchDesigner-compatible blob tracker. Carves a configurable depth slab out of the LiDAR scan, finds connected regions of in-range pixels, assigns each one a stable ID across frames, and streams per-blob metadata over OSC. Replaces a Kinect + Blob Track TOP chain with a single iPhone — no background plate, no lighting calibration, works on a moving camera. Requires LiDAR. See **Blob Tracking** below for full details.

### Transcription

Live on-device speech-to-text powered by iOS 26's `SpeechAnalyzer` framework. Camera view is replaced with a 200-bar mirrored waveform driven by the microphone; recognized words appear as captions; transcript streams over OSC, TCP, UDP, and NDI. Fully offline. Works on every iPhone. See **Transcription** below for full details.

### Motion

Live device IMU as scrolling graph lanes — accelerometer (X/Y/Z), gyroscope (X/Y/Z), compass heading, barometric pressure / relative altitude. Each enabled sensor draws its own lane and streams over OSC. Update rate selectable 30 / 60 / 100 Hz. Works on every iPhone.

### Audio

Real-time microphone audio analysis — bass / mid / high levels with rolling auto-gain, beat onset detection per band, dynamics burst trace, and a 20-band log-spaced FFT spectrum. Each enabled signal draws a scrolling lane and streams over OSC. Update rate selectable 30 / 60 Hz. Works on every iPhone.

### Switching Modes

Use the mode dropdown or say "switch to Depth" with Voice Control enabled. Switching is instant and does not interrupt an active stream.

---

## Streaming and Local Recording (Middle Page)

The middle page has a single iOS Camera-style shutter button at the bottom-center. The **STREAM | RECORD** segmented control directly below picks which paradigm the shutter triggers. The two are mutually exclusive: while one is running the other is locked, so you can't accidentally start a recording mid-stream.

### STREAM

Taps the shutter to start / stop streaming. All enabled transports send simultaneously.

Available transports:
- **TCP/UDP** — H.264 video (Color / Mono / Depth visual modes) or raw Float32 depth maps (Depth / Point Cloud modes)
- **NDI** — Industry-standard video-over-IP. Auto-discovered as `LOTA - <deviceLabel>` by TouchDesigner, OBS, vMix, Resolume, NDI Studio Monitor
- **OSC** — Camera pose, tracking, IMU, audio, transcription messages over UDP
- **PLY** — Live point cloud over TCP (binary, fixed wire format)

### RECORD

Taps the shutter to start / stop a local recording of the live composited Metal output. Encoded as H.264 `.mov`, saved to the **Photos** library via add-only access. Available in every camera-using mode (Color, Mono, Depth, Neural Depth, Point Cloud, Blob Track); disabled in Transcription / Motion / Audio (no camera content). Hard 4-hour cap per recording.

### Status Bar / Pill

The floating status bar pill at the top opens **Transmission Settings**, which contains: **This Device** (the iPhone's live IPv4 addresses with friendly Wi-Fi / USB Hotspot / Ethernet labels — tap to copy), **Receiver** (IP + Device Label), **Transport** (TCP/UDP), **NDI**, **OSC**, **Point Cloud Stream** (PLY), **Protocol Info**, **Help** (Replay Tutorial), and **Acknowledgements** (NDI® attribution).

While streaming, the per-protocol chips go from dim white to red so you can confirm at a glance what's live. A dim table at the bottom of the screen lists each active transmission with its `<host>:<port>` (and the NDI source name `LOTA - <deviceLabel>`), plus the iPhone's own active IPs.

### Wired vs. Wireless

The shutter's inner color reflects connection state. The transmit-button glyph swaps to a personal-hotspot icon when the configured Receiver IP routes through the iPhone's USB Hotspot subnet, so a wired direct phone↔Mac setup is unambiguous.

### Bottom-Bar Pinning in Landscape

The shutter + segmented control stay pinned to the device's *physical* bottom edge (where they sit in portrait), like the iOS Camera shutter. Rotate the phone and the island slides to whichever screen edge corresponds to that physical position. All other UI rotates naturally with the interface.

### Mode-Specific Settings

Every mode with tunable parameters shows a `<Mode> Settings` button just below the status bar (e.g. **Depth Settings**, **Blob Track Settings**, **Audio Settings**). Color and Mono show a dimmed "No settings for this mode" pill.

---

## Blob Tracking

Select Blob Track from the mode picker to turn the iPhone into a wireless TouchDesigner-compatible blob tracker. Requires LiDAR.

### While Active

- A `N blob(s)` count appears in the status bar with a hex-grid icon
- The current depth range (e.g. `0.5m – 3.0m`) is shown under the mode picker
- Each detected blob gets a 1-pixel hairline rectangle drawn over the camera feed
- The full composition (camera + outlines + optional ID labels) is captured by NDI so receivers see exactly what the phone shows
- Connected-components labeling runs off the ARKit delegate thread

### Base Styles

In Blob Track Settings → Detection:
- **Color** (default) — Live color camera feed
- **Mono** — Grayscale camera feed
- **Mask** — Grayscale subject silhouette on black
- **Binary** — Pure white silhouette on black (authentic TouchDesigner Blob Track TOP look)

Toggle `Draw Blob Bounds` off to see only the base layer. `Show ID Labels` draws `#1`, `#7`, etc. labels just outside the top-right of each bbox.

### OSC Addresses (TD-Compatible)

Field names match TouchDesigner's `blobtrackTOP_Class` verbatim. Every message is padded to 10 slots so CHOP channel counts stay stable — empty slots have `id == 0`.

- `/lota/blob/count` — active blob count (int)
- `/lota/blob/ids` — 10 stable tracker IDs
- `/lota/blob/u`, `/lota/blob/v` — normalized centroid (10 floats each)
- `/lota/blob/width`, `/lota/blob/height` — normalized bbox size (10 floats each)
- `/lota/blob/tx`, `/lota/blob/ty` — pixel centroid in 256×192 depth space (10 ints each)
- `/lota/blob/age` — seconds tracked (10 floats)
- `/lota/blob/state` — `0=new, 1=revived, 2=lost, 3=expired` (10 ints)

### Lifecycle States

A blob fires `state = 0 (new)` on first detection, stays silent while actively tracked, transitions to `lost` if it disappears, gets the same ID back via `revived` if it reappears within Revive Distance during the Revive Time window, otherwise transitions to `expired` and is permanently dropped. Mirrors TouchDesigner's blob lifecycle exactly.

### TCP/UDP Fallback

In Blob mode, TCP/UDP carries raw `Float32` depth maps (the same format as Depth mode), so receivers that want client-side analysis get the raw data. Structured per-blob metadata is OSC-only.

---

## Transcription

Select Transcription from the mode picker to turn the iPhone into a wireless live speech-to-text source. Camera view is replaced with a black canvas and a 200-bar mirrored waveform driven by the microphone. Recognized words appear as live captions. Uses iOS 26's on-device `SpeechAnalyzer` — fast, private, offline. Works on every iPhone.

### First-Time Setup

Microphone and Speech Recognition are requested upfront on first launch; the on-device speech model for your language is prefetched in the background, so transcription is usually ready the first time you open this mode. If you denied either permission, this mode displays a shortcut to iOS Settings. If the prefetch didn't finish, the model installs lazily on first use as a fallback.

### While Active

- A pulsing microphone icon and "LISTENING" label in the status bar
- The waveform pulses with your voice in real time and streams through NDI as a black-and-white video source
- Recognized words appear as large centered captions as they arrive
- Camera position OSC is suppressed so speech messages stand out
- Leaving transcription mode stops the audio engine — no background mic access

### OSC Addresses

- `/lota/speech/word` (string) — each recognized word as it arrives
- `/lota/speech/word_count` (int) — incrementing counter, visible in OSC In CHOP for signal flow monitoring
- `/lota/speech/partial` (string) — running partial transcript
- `/lota/speech/final` (string) — finalized sentences when the recognizer commits

String messages arrive in TouchDesigner's OSC In DAT (not OSC In CHOP — CHOP only handles numeric channels). `word_count` is the only speech message visible in OSC In CHOP.

### TCP/UDP Binary Wire Format

For custom integrations, recognized speech is also sent as binary frames over TCP and UDP — `FrameType.speechText = 4`, a 24-byte FrameHeader + UTF-8 payload. Header's `width` is repurposed for speech kind (0=word, 1=partial, 2=final), `height` for word index. Use the **LOTASpeechTCP** / **LOTASpeechUDP** TouchDesigner components below.

---

## Motion and Audio Modes

The Motion mode renders accelerometer X/Y/Z, gyroscope X/Y/Z, compass heading, and barometric pressure + relative altitude as scrolling graph lanes. Each enabled sensor draws its own lane and streams over OSC.

The Audio mode renders bass / mid / high levels (with rolling auto-gain), per-band beat onsets, a dynamics burst trace, and a 20-band log-spaced FFT spectrum across 20–20,000 Hz as scrolling graph lanes. Each enabled signal draws a scrolling lane and streams over OSC.

### Motion OSC Addresses

- `/lota/motion/accel/x`, `/y`, `/z` — gravity-removed acceleration in G-force
- `/lota/motion/gyro/x`, `/y`, `/z` — rotation rate in rad/s
- `/lota/motion/heading` — magnetic heading in degrees (0–360)
- `/lota/motion/pressure` — atmospheric pressure in kPa
- `/lota/motion/altitude` — relative altitude in meters

Update Rate: 30 / 60 / 100 Hz. 30 Hz matches ARKit; 100 Hz is useful for latency-critical controllers. Compass requires Location permission. Each sensor toggles independently and adds / removes its graph lane within ~50 ms (debounced).

### Audio OSC Addresses

- `/lota/audio/bass`, `/lota/audio/mid`, `/lota/audio/high` — continuous 0–1 energy
- `/lota/audio/drums/low`, `/lota/audio/drums/mid`, `/lota/audio/drums/high` — binary 0/1 per onset
- `/lota/audio/burst` — fast transient detector, pulse-shaped 0–1
- `/lota/audio/fft/0` through `/lota/audio/fft/19` — 20 log-spaced FFT bands, each normalized 0–1, rendered with a red→blue gradient

Update Rate: 30 / 60 Hz. Analysis runs internally at ~86 Hz and decimates to the chosen output rate. LOTA can only analyze the mic input — iOS sandboxing prevents capturing system audio from other apps.

---

## ARKit Tracking (Swipe Left)

Motion capture via ARKit, streamed as OSC data.

### Tracking Modes

| Mode | Camera | What It Tracks | OSC Addresses |
|------|--------|---------------|---------------|
| **Body** | Rear | 3D skeleton (18 key joints from 91 ARKit joints) | `/lota/body/skeleton`, `/lota/body/root`, `/lota/body/detected` |
| **Face** | Front | 52 facial blend shapes | `/lota/face/<blendShapeName>` (52 individual addresses as an OSC bundle) |
| **Hands** | Rear | 21 landmarks per hand, up to 2 hands | `/lota/hand/{left\|right}/{finger}` organized by finger |

### Overlays

- **Body** — White bones + green joint dots
- **Face** — Bar graph of the top 8 most active blend shapes
- **Hands** — Bone chains + joint dots (teal = left, orange = right)

Overlays can be toggled in Settings → Tracking.

### Hand Coordinate Modes

- **2D (default)** — Normalized screen-space coordinates. Works on every iPhone
- **3D (opt-in)** — World-space coordinates projected via LiDAR depth. Requires LiDAR. Toggle in Settings → Tracking → 3D Hand Coordinates

### Body / Face / Hand Requirements

- **Body** requires A12 chip or later
- **Face** requires TrueDepth camera (iPhone X or later)
- **Hands** works on every iPhone (Vision framework)

---

## Streaming Protocols

Four protocols, all independently configurable, all running simultaneously.

### NDI

Industry-standard video-over-IP. LOTA appears as `LOTA - <deviceLabel>` on your network and is auto-discovered by any NDI receiver. No IP configuration needed. The **Device Label** (random 4-char base36 default, customizable 4–20 chars) lets multi-device venues distinguish each phone's NDI source in TouchDesigner / OBS / NDI Studio Monitor. Label changes take effect within ~1 s without restarting the stream. When Neural Depth is enabled on a non-LiDAR phone, the side-by-side NDI toggle uses Neural Depth as the right half so multi-pane workflows work on every iPhone.

### TCP / UDP

H.264 video for visual camera modes (Color, Mono, Depth visual, Neural Depth, Point Cloud, Blob Track), raw `Float32` depth maps for Depth and Point Cloud modes. Configurable host, port, protocol.

### OSC

Camera pose, body / face / hand tracking, IMU, audio analysis, and transcription messages over UDP.

- `/lota/camera/position` (x, y, z) at ~30 Hz
- `/lota/camera/rotation` (quaternion) at ~30 Hz
- `/lota/camera/euler` (pitch, yaw, roll) at ~30 Hz
- `/lota/mode` and `/lota/fps` at 1 Hz
- Body / face / hand / motion / audio / speech addresses listed above

### PLY (live point cloud)

Live point cloud frames over TCP. Wire format is binary-only as of v1.2.3: `UInt32 LE point count` followed by `N × (3 Float32 XYZ + 3 UInt8 RGB)` = 15 bytes per point. Drop in **LOTABinaryPLYRecieverV2.tox** from the TouchDesigner section for plug-and-play receive.

### How to Start Streaming

1. Open Transmission Settings (status bar pill). Enable the protocols you need; set Receiver IP, ports, and Device Label. NDI requires no IP config.
2. Make sure both devices are on the same local network. 5 GHz Wi-Fi recommended for lowest latency; USB Hotspot (Personal Hotspot via Lightning / USB-C → Mac) gives a wired direct link.
3. Tap the shutter in **STREAM** mode. All enabled protocols start simultaneously. The status bar chips switch from dim white to red.

---

## TouchDesigner Integration

LOTA ships drop-in TouchDesigner components for each of its non-NDI transports.

### NDI Input (Easiest)

Drop an NDI In TOP into your project. LOTA appears in the source list automatically as `LOTA - <deviceLabel>`. Select it; you're receiving live video.

### LOTAPoints.tox — Drop-in Point Cloud Component

Ready-made TouchDesigner component that receives live PLY point cloud data. Drop it in; works out of the box. Default port 9848. Download: https://lidarota.app/LOTAPoints.tox

### LOTABinaryPLYRecieverV2.tox — Binary Point Cloud Receiver

High-performance binary point cloud receiver. Uses numpy bulk parsing, Script TOP textures, and GPU instancing to handle 49K+ points at 60 fps with ~1–2 ms Python overhead. Enable **PLY Streaming** in Transmission Settings → Point Cloud Stream to use it. Download: https://lidarota.app/LOTABinaryPLYRecieverV2.tox

### LOTASpeechTCP.tox — Speech TCP Receiver

TCP/IP DAT with a callback script that parses LOTA's binary speech frame format and writes recognized words, partials, and finals to a `speech_log` Table DAT. Handles stream buffering for frames split across TCP packets. Download: https://lidarota.app/LOTASpeechTCP.tox

### LOTASpeechUDP.tox — Speech UDP Receiver

UDP In DAT in "One Per Message" mode with a callback script that parses the same binary speech frame format. Simpler than TCP — each datagram is a complete frame. Low-latency fire-and-forget delivery. Download: https://lidarota.app/LOTASpeechUDP.tox

### Camera Tracking via OSC

Add an OSC In CHOP, set the port to match LOTA's OSC port. You'll receive camera position (x, y, z), rotation (quaternion), and euler angles at 30 Hz — perfect for driving a virtual camera or triggering effects from device movement.

### Audio Mode → Channel Operators

Drop an OSC In CHOP, point it at the phone's IP + OSC port. Channels `bass`, `mid`, `high` appear and start pulsing. Toggle Beat Detection on → `drums_low`, `drums_mid`, `drums_high` appear as clean 0/1 signals you can route into a Trigger CHOP for beat-synced effects.

---

## Gaussian Capture Page (Swipe Right)

This page records datasets for Gaussian Splatting and 3D reconstruction training, captures **PBR material** sets from flat surfaces, exports **IMU and Audio Trace** files, and (added in v1.2.6) lets you **inspect any saved `.ply` point cloud** in an interactive 3D viewer. During 3D-scene recording, a mesh wireframe overlay shows scanned surfaces building up in real time. During material capture, the page swaps to a plane-lock + shutter UI. In PLY Viewer mode, the page swaps to a dark canvas with orbit controls and a neon axis gizmo.

### Setup

1. **Choose export format** — Tap the format picker at the top
2. **Set export folder** — Tap the folder icon (bottom left) to pick an iCloud Files folder
3. **Tap record** — The red circle button starts capture

### Export Formats

| Format | Output | Best For |
|--------|--------|----------|
| **COLMAP** | `sparse/0/cameras.bin`, `images.bin`, `points3D.bin` + `images/` JPEGs | 3D Gaussian Splatting, OpenSplat, gsplat |
| **Nerfstudio** | `transforms.json` + `images/` JPEGs + `points3D.ply` | Nerfstudio/splatfacto, Instant-NGP |
| **Nerfstudio + Depth** | Same as Nerfstudio + `depth/` 16-bit PNG depth maps | Depth-supervised training (best geometry on flat surfaces) |
| **Point Cloud** | Single `points3D.ply` file (binary or ASCII) | Blender, CloudCompare, MeshLab |
| **Material** | PBR map set ZIP — `basecolor.png`, `normal.png`, `height.png` (16-bit), `ao.png`, `roughness.png`, `preview.png`, `material.json` | Substance Designer/Painter, Blender, Unreal, Unity, TouchDesigner |
| **IMU Trace** | `recording.csv` or `.tsv` + `manifest.json` + `README.txt` (+ optional `report.pdf`) | Motion analysis, sensor research, data science workflows |
| **Audio Trace** | Same shape as IMU Trace, with audio analysis columns | Music analysis, beat detection research, audio reactive workflows |
| **PLY Viewer** *(new in v1.2.6)* | Read-only — no output | Inspect any saved `.ply` point cloud on the device (LOTA captures or files dragged in via the Files app) |

Every export ZIP is named `LOTA_<timestamp>_<MODE>.zip` (e.g. `LOTA_2026-04-30_14-22-08_IMU.zip`) so the format is identifiable from the filename alone. PLY Viewer is the one exception: it loads files instead of writing them.

### What Happens During Recording

1. **Camera feed + mesh overlay** — You see the live camera with ARKit's scene reconstruction mesh building up as semi-transparent triangles (cyan near, purple far)
2. **Keyframe selection** — Not every frame is saved. Only frames where the camera moved at least 5 cm or rotated ~5° are kept
3. **Blur detection** — Motion-blurred frames are automatically rejected via Laplacian variance analysis
4. **Focus lock** — Autofocus disabled during recording to keep camera intrinsics consistent
5. **Haptic feedback** — A subtle tap each time a keyframe is captured
6. **Counters** — Elapsed time, keyframe count, total point count

### Capture Settings (new in v1.2.6)

The 3D-scene recording formats (COLMAP, Nerfstudio, Nerfstudio + Depth, Point Cloud PLY) share a **Capture Settings** sheet, opened via the settings pill below the format picker:

- **Capture Rate** (30 Hz default / 15 Hz / ~7 Hz) — iPhone LiDAR runs at 30 Hz. LOTA filters duplicate-buffer ARFrames first (ARKit hands back the same depth buffer on every other 60 Hz callback), then strides on the unique scans, so the picker reflects real LiDAR scans per second. 30 Hz keeps every scan, 15 Hz every other, ~7 Hz every fourth. Lower rates skip per-pixel unprojection and JPEG encoding for the dropped scans, roughly halving or quartering the captured point count and zip size. Useful for scanning a small subject without needing every scan, or for throttling a long scan to keep the device cool
- **PLY Format** (Binary default / ASCII; new in v1.2.7) — shown only when the Point Cloud format is selected. Binary (`binary_little_endian`) is compact and read natively by OpenSplat, COLMAP, MeshLab, and CloudCompare. ASCII writes a human-readable text PLY (one `x y z r g b` line per point), easier to inspect or script against but roughly 3× larger before zipping; ASCII artifacts are named `LOTA_<timestamp>_PLY_ASCII.zip`

### Material Capture

Select **Material** from the format picker to capture a flat surface as a complete PBR material set. The page UI swaps from the recording flow to a plane-lock + shutter flow. One tap, ~½-second bake, ZIP saved to your iCloud folder. Requires LiDAR.

**When to use this vs other formats**:

| You want… | Pick |
|---|---|
| A 3D scan of an object/room you can move around | COLMAP / Nerfstudio / Point Cloud |
| A tileable PBR texture of a flat surface (floor, wall, table, fabric, brick…) | **Material** |

**Capture flow**:

1. Pick Material from the format picker — page swaps to the material-capture UI
2. Pick an export folder via the folder icon if you haven't already
3. (Optional) Tap **Material Settings** for output resolution, AO sample count, normal convention, delight strength, or roughness scale
4. Point at a flat surface — status chip changes to "Surface detected — tap Lock Plane"
5. Tap **Lock Plane** — auto-installs a 20 cm screen-aligned square patch centered on what you're pointing at; shutter enables
6. Tap the shutter — torch fires briefly (flash-pair sequence ~200 ms) → bake (~220 ms at 1024² on iPhone 15/16 Pro)
7. **Material Save Summary** sheet appears with the live PBR sphere preview, file-size estimate, and metallic toggle
8. Tap **Save to Files** — ZIP written to your iCloud folder, brief "Sent" confirmation
9. Plane lock stays active; shoot another patch immediately

**Output ZIP contents** (`LOTA_Material_YYYY-MM-DD_HH-MM-SS.zip`):

| File | Color space | Bit depth | What it is |
|---|---|---|---|
| `basecolor.png` | sRGB | 8-bit | Albedo / diffuse, white-balanced and de-lit via flash-pair specular subtraction |
| `normal.png` | linear | 8-bit | Tangent-space normal from LiDAR depth gradient (OpenGL +Y up by default; DirectX selectable in Material Settings for Unreal) |
| `height.png` | linear | **16-bit** | Plane-relative height, ±25 mm range mapped to UInt16 |
| `ao.png` | linear | 8-bit | Horizon-based ambient occlusion baked from the height map |
| `roughness.png` | linear | 8-bit | Per-texel estimate from flash-pair specular sharpness × user roughness scale |
| `preview.png` | sRGB | 8-bit | Cook-Torrance BRDF sphere render — what the material looks like under simple lighting |
| `material.json` | — | — | Capture metadata: device + LiDAR generation, patch size, normal convention, tiling hint for a 1 m² target, per-channel descriptors |

The key field is `patch.planeMeters` — the physical real-world size of the captured patch. `tilingHintFor1mSquare` is the precomputed repeat-count to tile this material across 1 m² in your DCC.

**Receiver workflows**:

- **Substance Designer**: drop the unzipped folder onto a new graph; Substance auto-creates a Material node with all six channels wired
- **Substance Painter**: drag the folder into Shelf → Materials, apply to a mesh
- **Blender**: add an Image Texture node per file in the Shading editor, wire to a Principled BSDF, set non-color spaces on `normal`, `roughness`, `ao`, `height`
- **Unreal Engine**: drop the folder into Content Browser → auto-creates a Material Instance. Switch normal convention to **DirectX** in Material Settings before capture
- **Unity**: standard URP/HDRP Lit shader inputs map 1:1
- **TouchDesigner**: load each PNG into a Movie File In TOP, wire to a PBR MAT

**Tips for best results**:
- Hold the phone still during the shutter tap (~200 ms flash-pair window)
- Capture from 20–60 cm at the default 20 cm patch size
- Ambient + iPhone torch lighting works best — too-bright ambient washes out the controlled flash signal
- Best surfaces: wood, fabric, leather, brick, concrete, plaster, painted drywall, asphalt, ceramic tile (matte), unpolished plastic, paper, carpet, bark, stone

**Limitations** (the pipeline degrades or fails on these):

| Surface | What goes wrong |
|---|---|
| Glass / transparent | LiDAR passes through; basecolor captures whatever's behind |
| Mirrors / chrome | Captures the reflected scene, not the surface itself |
| Polished metal | Basecolor contaminated by reflections |
| Skin / wax / translucent plastic | Subsurface scattering not modeled |
| Velvet / fur | Anisotropic / sheen BRDFs not modeled |
| Wet surfaces | Captures the wet film, not the dry substrate |

**v1.2 limitations** (planned for v1.3+): metallic is a uniform toggle (not per-texel); roughness is a flash-pair heuristic; framing rect is auto-installed at 20 cm; single-shot only.

### IMU Trace and Audio Trace

Pick **IMU Trace** or **Audio Trace** from the format picker to record sensor or audio analysis data over time as a structured file for a data science / motion / music workflow. The right page swaps to the matching live scrolling-lane visualization while the trace is being collected.

**Capture flow**:

1. Pick **IMU Trace** or **Audio Trace** from the format picker
2. Set the export folder (folder icon, bottom left) if you haven't yet
3. Open the **IMU Trace Settings** / **Audio Trace Settings** sheet — pick channels, file format (CSV / TSV), bundle a PDF report, optional auto-stop duration cap (5–600 s)
4. Tap the shutter to start — sample counter + elapsed timer update at 10 Hz
5. Tap again to stop (or let the cap auto-stop). Saves to your iCloud folder; a save-summary sheet shows file size and sample count

**ZIP contents**:

| File | Always included | Description |
|------|-----------------|-------------|
| `recording.csv` (or `.tsv`) | Yes | One row per sample. First column `t_seconds`; others described in the manifest |
| `manifest.json` | Yes | Device (consumer name, e.g. `iPhone 17 Pro Max`), version, timestamps, sample count, sample rate, stop reason, per-column dictionary (name, unit, dtype, description) |
| `README.txt` | Yes | One-paragraph human description of the bundle layout |
| `report.pdf` | Optional toggle | Multi-page dark-theme report with cover, per-channel charts, statistics table |

**PDF report contents** (when enabled):

| Mode | Pages |
|------|-------|
| **IMU Trace** | Cover · Accelerometer X/Y/Z line · |A| magnitude · Gyroscope X/Y/Z line · Compass heading polar rose (if heading recorded) · Pressure & altitude (if pressure recorded) · Per-channel statistics |
| **Audio Trace** | Cover · FFT spectrogram (20 bands × time, viridis heatmap) · Bass/Mid/High levels · Beat events timeline (per band) · Dynamics burst trace · Per-channel statistics |

Charts that have no data to plot show a centered "No data captured" placeholder so the recipient can tell an empty page is intentional, not broken (typical for an Audio Trace recorded in a silent room — the FFT charts populate, the beat-events page shows the placeholder).

**When to use these**:
- Capture a short walk to plot device motion against ground-truth GPS in a Jupyter notebook
- Record a song through the mic to extract beat onsets for an Ableton MIDI workflow
- Sample IMU data from a moving rig and ship it to TouchDesigner / Blender as a baked animation source
- Build a training dataset for sensor-fusion or audio-classification models

### PLY Viewer (new in v1.2.6)

Select **PLY Viewer** from the format picker to open the read-only inspector. The format picker pauses ARKit (the live camera goes black), and the right page swaps to a dark canvas with the bottom shutter morphed into a folder glyph. Tap the folder to open the file picker (filtered to `.ply` via the new `dev.lota.ply` UTI), choose any PLY on the device (LOTA captures, iCloud Drive, third-party sources), and the file loads off the main thread. The viewer handles two kinds of `.ply`: a plain point cloud renders as flat points, while a 3D Gaussian Splat file is detected automatically and rendered as splats (new in v1.2.7).

**Once a cloud is loaded**:

- A centered top **info chip** shows point count (or a splat count for a Gaussian splat), AABB extents (in metres or centimetres), file size, and badges for `subsampled` (yellow) and `no RGB` when applicable
- A small **neon axis gizmo** pins to the top-trailing corner. Each axis cap is a 3D-projected dot (positives carry X/Y/Z letters, negatives are hollow rings, Blender convention) and updates live as you orbit. Tap any cap to spring-snap the camera to that view
- The bottom shutter morphs to a **load-new-file** button so you can swap clouds without leaving the mode

**Gestures**:

| Gesture | Action |
|---|---|
| One-finger drag | Orbit yaw / pitch |
| Two-finger drag | Pan the orbit target |
| Pinch | Zoom (clamped to 0.05× / 8× of fit-bounds distance) |
| Double-tap | Reset to fit-bounds with a 15° downward pitch |

**PLY Viewer Settings** (cog below the format picker):

| Section | Controls |
|---|---|
| **Color** | Color Mode (Original / Solid / By Height / By Distance / By Axis), Solid color picker, palette grid for scalar modes (same nine LiDAR depth palettes), axis segmented (X/Y/Z) for By Axis |
| **Orientation** | Camera Mode (Free Orbit / Auto Spin / Top / Front / Side), spin speed slider (0.05–2.0 rev/s) when Auto Spin is selected, Y-up / Z-up segmented, per-axis Mirror toggles, Reset View button |
| **Point Size** | Slider 1.0–12.0 |

All settings persist between launches. When the loaded file is a **Gaussian splat**, the Color section and Point Size slider are replaced by a single **Splat Size** slider (0.25×–3.0×, default 1.0×).

**Gaussian splats** (new in v1.2.7):

When the loaded file is a 3D Gaussian Splat — the layout OpenSplat and other 3DGS trainers write — the viewer detects it from the header and renders it with proper splatting: each Gaussian is an oriented, anisotropic, alpha-blended ellipse rather than a flat dot, depth-sorted on the GPU so it composites correctly from every angle.

- The info chip shows a **splat count** instead of a point count
- The settings sheet replaces the **Color** section and **Point Size** slider with a single **Splat Size** slider (0.25×–3.0×, default 1.0×). Below 1.0× thins the splat to reveal structure; above 1.0× fills gaps in a sparse splat. Splats carry their own baked color, so the point-cloud color modes do not apply
- **Orientation** (camera mode, spin, Y-up / Z-up, mirrors, Reset View) works exactly as it does for a point cloud. OpenSplat results are not always Y-up — if a splat loads tipped on its side, flip Y-up / Z-up first
- View-dependent shading (specular glints from higher-order spherical harmonics) is not rendered yet, so the splat can look slightly flatter than it does in a desktop viewer
- Capped at **2,000,000 Gaussians**. Larger splats are uniformly sub-sampled and the chip shows the `subsampled` badge

**Color modes in detail**:

- **Original** — per-point RGB from the file. Falls back to white if the file has no color
- **Solid** — single user-picked color, useful for inspecting shape without color noise
- **By Height** — Y coordinate mapped through the selected palette
- **By Distance** — distance from the camera to each point, recomputed live as you orbit. Good for spotting depth structure
- **By Axis** — same as By Height but along any of X / Y / Z. Lets you find the long axis of an asymmetric scan

**Limits and behavior**:

- **5 M point cap.** Larger files are uniformly sub-sampled and the chip shows a `subsampled` badge. A 5 M-point cloud needs ~120 MB of GPU buffers
- **Binary little-endian PLY only.** ASCII and big-endian PLY produce a clear "Unsupported PLY format" error
- **Extra properties** (normals, alpha, intensity) are tolerated. The parser builds an offset map for whatever the file declares, so it can read clouds from Blender, CloudCompare, Nerfstudio, and similar tools as long as `x y z` (and optionally `red green blue`) are present
- **Mirror toggles** apply at draw time (model matrix), not at parse time. The axis gizmo stays anchored to world axes regardless of mirror state
- **ARSession resumes** as soon as you switch off the PLY Viewer format. Swiping between pages while a cloud is loaded keeps it in memory

**Tips**:
- For files saved from Blender or CloudCompare, the **Y-up / Z-up** setting is the first thing to check if the cloud appears tipped on its side. LOTA's own captures are Y-up
- **By Distance** color mode is useful as a quick "is this cloud noisy from the inside?" inspection — noise looks like a fog when colored by distance from the orbit center
- **Auto Spin** at ~0.2 rev/s is a good speed for casually presenting a capture without manual orbiting

---

## Accessibility

LOTA is designed to meet Apple's App Store accessibility standards. Every feature works for every user.

### VoiceOver

Every control is labeled and announced. Mode switches, streaming state, and recording status are all spoken aloud.

### Voice Control

Say "stream", "record", or "switch to Depth" and LOTA responds. Every button is discoverable by voice.

### Dynamic Type

Text scales up to 200%+. The UI reflows to a single-column layout at extreme sizes so nothing gets cut off.

### Dark Interface

Designed dark from the start. Every screen, menu, and control uses a true dark color scheme.

### Differentiate Without Color

Status indicators swap to distinct symbols when this setting is enabled. Shapes, icons, and text labels replace color as the sole differentiator.

### Increase Contrast

Swaps blur materials for solid backgrounds and boosts status colors for guaranteed readability over any camera feed.

### Reduce Motion

All transitions respect the system Reduce Motion setting. Visual feedback stays, decorative animation goes.

---

## Frequently Asked Questions

### Which iPhones support LOTA?

Any iPhone running iOS 26.2 or later. LiDAR features (Depth, Point Cloud, Blob Track, Material Capture, LiDAR-based 3D-scene recording formats, 3D hand coordinates) require iPhone 12 Pro or later. Color, Mono, **Neural Depth** (Depth Anything V2 on the Apple Neural Engine), Transcription, Motion, and Audio modes, NDI streaming, and TCP/UDP streaming all work on iPhones without LiDAR — and Neural Depth doubles as the side-by-side NDI fallback so multi-pane workflows work on every iPhone. Face tracking requires TrueDepth camera (iPhone X or later). Body tracking requires A12 chip or later. **LOTA is officially iPhone-only.** iPad Pro models with LiDAR have run successfully during beta testing, but iPad is not an officially supported target — your mileage may vary.

### Does Transcription mode need internet access?

No. Transcription uses iOS 26's on-device `SpeechAnalyzer` framework and runs entirely offline. After you grant Speech Recognition permission on first launch, LOTA prefetches the on-device speech model for your locale in the background, so by the time you open Transcription mode the model is on disk and first captions arrive without a download wait. If the prefetch didn't finish, the model installs lazily on first use. Audio never leaves your device.

### Do the receiving machine and iPhone need to be on the same network?

Yes. For TCP, UDP, OSC, and PLY streaming, both devices must be on the same local network. NDI also uses the local network but handles discovery automatically. A 5 GHz Wi-Fi connection is recommended. USB Hotspot (Personal Hotspot via Lightning / USB-C → Mac) gives a wired direct link if Wi-Fi is unreliable.

### What software can receive LOTA streams?

Any NDI-compatible software (TouchDesigner, OBS, vMix, Resolume), any tool that reads TCP/UDP sockets, and any OSC-capable application (Max/MSP, Ableton, Unreal Engine).

### How do I use the export data for Gaussian Splat training?

Export a COLMAP or Nerfstudio dataset from the Gaussian Capture page, transfer the zip to your training machine, and point OpenSplat, Nerfstudio, or gsplat at the extracted folder. The files are in the exact format these tools expect.

### Is there a latency cost to streaming?

NDI and TCP/UDP streams typically add 1–3 frames of latency depending on your network. OSC tracking data arrives at 30 Hz with sub-frame latency. A wired (USB Hotspot) connection or 5 GHz Wi-Fi keeps things as fast as possible.

### What export format should I use?

COLMAP for OpenSplat and gsplat. Nerfstudio for splatfacto and Instant-NGP. Nerfstudio + Depth for best geometry on flat or featureless surfaces. Point Cloud (PLY) for quick visualization in Blender or CloudCompare. Material for PBR texture sets from flat surfaces. IMU / Audio Trace for time-series data science workflows. PLY Viewer for inspecting a saved point cloud on the device without exporting it.

---

## Privacy

LOTA does not collect, store, or transmit any personal data to remote servers. All sensor data is processed entirely on your device. Data only leaves your iPhone when you choose to stream it over your own local network, save a recording to your own Photos library, or export to your own iCloud storage.

- No analytics, crash reporting, or usage tracking in the app
- No user accounts
- Face, body, and hand tracking data processed entirely on-device, never sent to any remote server
- Neural Depth inference runs on the Apple Neural Engine, on-device
- Transcription uses Apple's on-device `SpeechAnalyzer`; audio never leaves the device
- NDI SDK contains no telemetry or external connections
- Website uses Vercel Analytics (anonymous, aggregated, no cookies)
- Newsletter emails stored by Resend, not shared with third parties

Contact: privacy@lidarota.app