NIST Emergency Response Robots program — standard test methods for aerial, ground, and aquatic response robots.
Robotics Research Engineer, Dec 2023 – Sep 2025 · Pathways Student / PREP Researcher, Jun 2020 – Dec 2023
Context
When a fire department or bomb squad buys a drone or ground robot, it needs a way to answer a plain question: what can this system actually do? NIST’s Emergency Response Robots program answers it with standard test methods — physical apparatuses, defined procedures, and quantitative scoring that make robot performance measurable, repeatable, and comparable across platforms and operators.
The program spans three domains — aerial (sUAS), ground, and aquatic response robots — and feeds its methods into formal standardization through the ASTM E54.09 committee on response robots. The same test lanes and scoring artifacts are exercised at national and international events, including ICRA, IROS, RoboCupRescue, UTAC, TXPSRS, and the PSCR UAS Challenges.
I worked inside this program for about five years (2020–2025): first as a PREP researcher and Pathways student supporting evaluation workflows, then as a Robotics Research Engineer developing test methods, apparatuses, scoring systems, and evaluation infrastructure end to end.
Role
Robotics Research Engineer (Dec 2023 – Sep 2025). Led test development and validation (V&V) for uncrewed systems — UAS, UGV, and ROV — producing repeatable protocols, measurable performance criteria, and regression-ready test suites. Designed and built physical test apparatus, developed remote evaluation frameworks and scoring systems, managed the UAV fleet, and authored the SOPs, JHAs, and technical documentation that let distributed teams run the same test the same way. Tracked anomalies and defects with clear reproduction steps and evidence, and communicated status across teams to drive resolution.
Early track (Jun 2020 – Dec 2023). Supported standardized evaluation workflows for uncrewed systems — test setup, data capture, scoring, and reporting. Built practical tools and scripts to streamline analysis and improve consistency across repeated test runs, and assisted with stakeholder coordination, logistics, and documentation for the program’s standards-focused work.
Method & System
Every test method in this program follows the same lifecycle, and most of my work sat somewhere on it:
- Operational need. Requirements come from responders, not from the lab. I worked directly with first responders — Montgomery and Prince George’s County Bomb Squads, ATF, USCG, and Capitol Police — to design training scenarios, select tests, and fold field feedback back into the methods.
- Apparatus. The need becomes a physical test apparatus: reproducible, transportable, and buildable from common materials, so any agency can fabricate the same lane and get comparable results.
- Metrics. Each apparatus carries defined scoring — completeness, accuracy, time, acuity — so a trial produces numbers instead of impressions.
- Validation. Methods are run repeatedly across platforms, operators, and sites; procedures and scoring are refined until results are repeatable enough to trust.
- Standardization. Mature methods move into the standards process. I contributed to ASTM E54.09 aerial and dexterity standards development, supported re-balloting preparation for the E3132/E3132M logistics practice, and contributed to ASTM/NATO standardization efforts on UAV performance, dexterity, and operator proficiency.
Technical Work
Test apparatus & confined-space navigation
Designed and implemented test apparatuses for UAV navigation in confined spaces using ISO shipping containers, improving measurement of maneuverability, aperture negotiation, and indoor mapping performance.
Led development of ultra-confined test requirements at 2-inch scale with ATF Special Operations, and created modular labyrinths — built from milk crates, joists, and floorboards — to emulate attics and crawlspaces for repeatable trials in the tightest spaces a robot is asked to enter.
Spearheaded the upgrade of the ATF drone test facility in West Virginia, designing and assembling apparatuses that support nationwide remote-pilot-in-command refresher courses and operator scoring training.
The apparatus work extended underwater as well: designed and iterated physical test apparatus in SketchUp CAD for underwater robot evaluation, progressing through at least two documented design versions (Underwater Standing Apparatus v5 → v6).
sUAS drop testing & instrumentation
Led and analyzed controlled sUAS drop tests to assess fall dynamics, impact damage, and survivability — including design of a carrier stopping-and-release mechanism and indoor drop dampers so trials stay repeatable and safe. Mentored SURF students on outdoor drop tests and robot operations, and maintained the shared documentation and safety systems (netted enclosures, signage, hazard assessments) for UAV and robot testing spaces.
Visual & thermal test artifacts
Created and maintained high-precision visual and thermal test artifacts — Landolt-C acuity targets, QR-coded markers, and thermal identification inserts — used to benchmark perception, operator-interface performance, and thermal acuity under controlled conditions. These artifacts are what turn “the camera seems good” into a measured acuity score.
Autonomy evaluation: HIL/SIL/MIL and 3D mapping vs. LiDAR ground truth
Applied hardware-in-the-loop, software-in-the-loop, and model-in-the-loop testing workflows to evaluate UAS 3D indoor mapping capability, comparing UAS-generated point clouds against ground-truth LiDAR scans (e.g., Leica BLK2Go) and analyzing SLAM drift and structural accuracy. Conducted autonomous and semi-autonomous flight tests indoors and outdoors, integrating metrics for LiDAR accuracy, thermal sensor validation, mapping completeness, and obstacle avoidance.
Remote evaluation frameworks & scoring
Developed remote evaluation frameworks and performance scoring systems for UAVs operating in open, obstructed, and confined test lanes, enabling asynchronous review of standardized video and telemetry by distributed teams. Implemented a simplified remote capture system and standardized video documentation workflow for drone and robot test trials — used by NIST and at international competitions (RoboCupRescue, ICRA, IROS) to enable remote proctoring. Supported cross-agency coordination (DHS, DOJ, APSA, bomb squads), remote course administration, and clear documentation for geographically dispersed teams.
Fleet management & Remote ID compliance
Managed the NIST UAV fleet across Blue/Green UAS classifications, Remote ID compliance, and airworthiness — performing maintenance, repairs, and battery management to keep aircraft ready for indoor and outdoor missions.
Completed a full inventory of the NIST drone fleet for audit, identified non-compliant aircraft, and implemented Remote ID compliance (modules and labeling) along with a safe LiPo battery disposal process.
Wildfire suppression test methods (with NFRL)
Co-led development of incipient wildfire suppression test methods integrating drones, thermal sensing, and test instrumentation, and led development of a wildfire simulation burn test with the National Fire Research Laboratory (NFRL) and XPRIZE partners to evaluate drone capabilities for incipient fire detection, inspection, and suppression under controlled wildfire-like conditions.
Co-led SURF students in designing and building the wildfire test apparatuses themselves — the “Fire Tree” and “Thermal Campfire Omni” — initiated the controlled-burn suppression collaboration with NFRL, and authored the burn-test hazard analysis paperwork that let the trials run.
This body of work became my first-author publication (see Evidence below).
Automated scoring on the edge: Jetson YOLO-OCR
Integrated YOLO-OCR pipelines on an NVIDIA Jetson Orin Nano to extract thermal readings and other indicators from UAV video quadrants, supporting automated scoring and situational awareness. Instead of a human pausing video to read a thermal target, the pipeline reads it from the frame — a step toward test trials that score themselves.
CAD-to-simulation: test lanes in Unreal Engine
Created true-to-scale CAD and SketchUp models of test lanes and arenas and integrated them into Unreal Engine and other simulation environments, enabling virtual familiarization flights and remote scenario evaluation before anyone flies the physical lane.
Event arena planning
Planned and modeled test arenas for national and international events — ICRA, IROS, RoboCupRescue, UTAC, TXPSRS, and the PSCR UAS Challenges — aligning layouts with NIST metrics, responder requirements, and logistics constraints.
Proposed and introduced slip-disc crossover-slope tasks that were later adopted into the ICRA and IROS Quadruped Challenges.
Sub-case-study 1 — Robot Evaluation in Operational Training Environments: DOJ & Guardian Centers, 2020–21
Overview
Between 2020 and 2021, NIST’s robot evaluation program participated in a series of large-scale operational testing events at professional training facilities. These exercises provided realistic, high-fidelity test environments far beyond typical lab conditions — enabling evaluation of ground robots in settings that closely mirror actual emergency-response deployments.
2020 DOJ procurement exercise — Poolesville, MD
The 2020 Department of Justice procurement exercise took place at the Montgomery County Police Department training center simulation environment in Poolesville, Maryland — a facility providing large-scale urban scenario environments used for law enforcement and emergency-response training. The exercise evaluated ground robot performance in this realistic operational setting, assessing mobility, navigation, and task execution relevant to first-responder and tactical applications. Course infrastructure included a tire-wall maze arena and bar-and-post gate obstacle apparatus, modeled in 3D as part of course design and documentation.
2021 UTAC exercise — Guardian Centers, Perry, GA
The 2021 UTAC exercise was held at Guardian Centers in Perry, Georgia — one of the most comprehensive operational training facilities in the United States, with extensive mock urban environments including structures, vehicles, and terrain at operational scale. Ground robots were evaluated across a range of tasks reflecting real-world emergency-response scenarios.
Why realistic environments matter for robotics testing
Laboratory test conditions, while essential for controlled measurement, cannot fully replicate the operational complexity of real environments. Large-scale training facilities like Guardian Centers and the Poolesville simulation site provide authentic terrain complexity, architectural variety, and mission-relevant task structure. Testing there produces performance data far more predictive of actual deployment outcomes than laboratory-only evaluation — which is exactly what test methods built for responders need to prove. This work supports the broader NIST effort to develop test methods and performance metrics that are valid, reliable, and relevant to actual emergency-response requirements.
My role: evaluation support across both exercises during my early NIST track — test setup and apparatus staging, data capture, scoring support, and standardized documentation.
Sub-case-study 2 — MCFRS UAS Challenge 2024: Autonomous Indoor Mapping
Event overview
The 2024 Montgomery County Fire and Rescue Service (MCFRS) UAS Challenge was a competitive evaluation event hosted at MCFRS facilities. It tested UAS platforms on a series of standardized indoor mapping and inspection tasks designed to simulate realistic emergency-response scenarios, run as a joint effort between NIST and the Public Safety Communications Research (PSCR) program.
Autonomous indoor mapping — Skydio
A key component of the challenge focused on autonomous indoor mapping using the Skydio UAS platform. The system executed fully autonomous flight runs through the MCFRS facility interior, generating AR-overlay mapping data and visual representations of the indoor environment. This evaluation context maps directly onto first-responder situational awareness: accurate, rapid indoor mapping supports search, rescue, and hazard assessment operations.
Testing & evaluation significance
The indoor mapping challenge assessed navigation stability, coverage completeness, obstacle avoidance, and mapping accuracy — the same metric family the NIST indoor mapping test methods formalize, and metrics that translate directly to operational requirements for emergency-response UAS deployment. Events like this are where standardized test methods meet live autonomous systems in front of the responders who will use them.
At a glance: Skydio UAS, autonomous flight mode · MCFRS facility, Montgomery County, MD (select imagery also captured at the NIST Robotics Building, Gaithersburg, MD) · evaluated 2024 · AR-overlay autonomous mapping documentation · standards connection: NIST indoor mapping performance metrics for emergency-response UAS evaluation.
My role: evaluation support — test-method-aligned scoring, observation of autonomous mapping runs, and standardized documentation.
Evidence
First-author publication:
Fraley, A., Nowery, E., Devine, M., Jacoff, A., Oh, P. — “Adapting NIST Aerial Drone Tests for Thermal Identification, Inspection and Suppression of Wildfires.” NIST, published December 6, 2024; presented at the 2025 IEEE 14th Annual Computing and Communication Workshop and Conference (CCWC 2025), Las Vegas, NV.
Public outreach library: produced 15+ distinct public-facing outreach materials (posters, trifolds, proficiency brochures) explaining NIST drone test standards to external audiences, 2016–2024 — a lower bound from the preserved materials themselves. A selection appears on the Publications & Outreach page.
Curriculum adoption: co-developed UAS test methods, scoring artifacts, and training-ready documentation adopted into the APSA Drone Train-the-Trainer curriculum — used by name, including international deliveries.
What This Demonstrates
- Full test-method lifecycle ownership — from a responder’s operational need to apparatus, metrics, validation, and formal standards work (ASTM E54.09, E3132/E3132M re-balloting support).
- Cross-domain evaluation range — aerial, ground, and aquatic systems; lab lanes, ISO containers, ultra-confined spaces, and full-scale operational training facilities.
- Instrumented, evidence-first testing — LiDAR ground truth, thermal and visual acuity artifacts, standardized video and telemetry capture, and scoring that distributed teams can review asynchronously.
- Applied software where it moves the test forward — YOLO-OCR automated scoring on Jetson edge hardware, CAD-to-Unreal simulation of physical test lanes, and practical analysis tooling.
- Documentation others can run without me — SOPs, JHAs, hazard analyses, scoring sheets, and training materials adopted into a named national curriculum.
- Published, citable output — a first-author NIST/IEEE paper grounding the wildfire test-method work.
Tools & Capabilities
Standard test-method development · operational T&E and V&V · scoring-system design · remote/distributed evaluation · SOP/JHA authorship · sUAS operations (FAA Part 107) · UAV fleet management & Remote ID compliance · drop testing · HIL/SIL/MIL · LiDAR ground truth (Leica BLK2Go) · thermal/visual acuity benchmarking · SketchUp CAD · Unreal Engine simulation · Python · YOLO-OCR on NVIDIA Jetson Orin Nano · ASTM E54.09 standards process · cross-agency coordination