diff --git a/docs/device/4-device-collector.md b/docs/device/4-device-collector.md index b4359fc9e..887d59df7 100644 --- a/docs/device/4-device-collector.md +++ b/docs/device/4-device-collector.md @@ -75,7 +75,7 @@ mod: listen_dirs: - /home/bag/ - # 当前时间距离文件更新时间超出 {skip_period_hours} 小时的时候，文件不会被监听/采集 + # 当前时间距离文件更新时间超出 {skip_period_hours} 小时的时候，文件不会被监听 skip_period_hours: 2 # 设备端的采集目录，作为项目中数据采集任务与规则采集的指定目录 @@ -83,6 +83,14 @@ mod: - /home/bag/ - /home/log/ + # 递归遍历所有子文件夹设置，针对监听目录和采集目录，是否遍历填写目录下的所有嵌套子文件夹，默认为否 + recursively_walk_dirs: true + + # 手动任务采集时指定上传额外文件，支持文件夹和文件，填写绝对路径；支持特定的时间变量模版 + additional_files: + - /home/just2004noetic/Downloads/testcase + - /additional_files/{{start_time.format('YYYY-MM-DD')}}/log/ + # 假设机器端存在 /home/coscene/device.yaml 文件，其内容为 # soft_version: v1.0 # @@ -124,7 +132,9 @@ collector: - 选填项 - 对应标识码字段名 -- `skip_period_hours`：若当前时间距离文件更新时间超出设定时间，文件不会被监听/采集。 +- `skip_period_hours`：若当前时间距离文件更新时间超出设定时间，文件不会被监听。 + +- `recursively_walk_dirs`：当监听和采集目录存在子文件夹，是否遍历嵌套子文件夹中的文件。默认为否，只读取当前文件夹下的一级所有文件 - `listen_dirs`： - 选填项 @@ -133,6 +143,11 @@ collector: - `collect_dirs`：设备端的采集目录，作为项目中数据采集任务与规则采集的指定目录。 +- `additional_files`： + - 选填项 + - 手动任务采集时，支持补充采集额外的文件信息（如地图、日志等）。支持填写文件夹以及文件绝对路径，文件夹会递归遍历所有的子文件 + - 路径支持模版变量信息，如 `/home/{{start_time.format('YYYY-MM-DD')}}/log/`, 模版变量在采集任务中会替换成对应的采集任务时间信息，依据选择的采集开始和截止时间修改，变量会绑定更新，渲染为对应的值。具体的语法请参考 [模板语法使用](#模板语法使用) + ```yaml mod: # mod 名称，默认 default，支持监听设备端指定目录下的文件，若有自定义的监听形式，请联系刻行时空 @@ -160,6 +175,14 @@ mod: collect_dirs: - /home/bag/ - /home/log/ + + # 会递归遍历 /home/bag/ 和 /home/log/ 目录下的所有子文件夹 + recursively_walk_dirs: true + + # 手动采集任务开始时间为 2025-06-27 14:00:00，额外补充上传 /home/just2004noetic/Downloads/testcase 和 /home/2025-06-27/log 目录下的所有文件，包含子文件夹 + additional_files: + - /home/just2004noetic/Downloads/testcase + - /home/{{start_time.format('YYYY-MM-DD')}}/log/ ``` ### 设备事件属性（device） @@ -189,6 +212,94 @@ topics: - error_code ``` +--- +### 模板语法使用 + +## 概述 + +模板语法是一种动态生成字符串的强大工具，特别适用于根据时间范围动态生成文件路径和文件名。基于 [Handlebars](https://handlebarsjs.com/) 模板引擎，集成了 [Day.js](https://day.js.org/) 时间处理库。 + +通过掌握这些模板语法，您可以轻松创建动态的文件路径和文件名，大大提高数据管理的灵活性和自动化程度。 + +## 基本语法 + +### 支持的变量 + +目前支持以下两个时间变量： + +- `start_time` - 当前采集任务的开始时间 +- `end_time` - 当前采集任务的结束时间 + +### 基本模板格式 + +``` +{{变量名.format('格式字符串')}} +``` + +### 示例 + +```javascript +// 基本用法 +'{{start_time.format("YYYY-MM-DD")}}' +// 输出: "2021-01-01" + +// 组合使用 +'data/{{start_time.format("YYYY")}}/{{start_time.format("MM")}}/file.log' +// 输出: "data/2021/01/file.log" +``` + +## 时间格式化 + +### 常用格式化选项 + +基于 [Day.js 格式化文档](https://day.js.org/docs/en/display/format)，支持以下格式： + +| 格式 | 输出 | 描述 | +|------|------|------| +| `YYYY` | 2021 | 四位年份 | +| `YY` | 21 | 两位年份 | +| `MM` | 01-12 | 月份（补零） | +| `M` | 1-12 | 月份 | +| `DD` | 01-31 | 日期（补零） | +| `D` | 1-31 | 日期 | +| `HH` | 00-23 | 小时（补零） | +| `H` | 0-23 | 小时 | +| `mm` | 00-59 | 分钟（补零） | +| `m` | 0-59 | 分钟 | +| `ss` | 00-59 | 秒（补零） | +| `s` | 0-59 | 秒 | + +### 格式化示例 + +```javascript +// 日期格式 +'{{start_time.format("YYYY-MM-DD")}}' // 2021-01-01 +'{{start_time.format("YYYY/MM/DD")}}' // 2021/01/01 +'{{start_time.format("MM-DD-YYYY")}}' // 01-01-2021 + +// 时间格式 +'{{start_time.format("HH:mm:ss")}}' // 10:30:00 +'{{start_time.format("HH-mm")}}' // 10-30 + +// 组合格式 +'{{start_time.format("YYYY-MM-DD HH:mm:ss")}}' // 2021-01-01 10:30:00 +'{{start_time.format("YYYYMMDD_HHmmss")}}' // 20210101_103000 + +// 按年月日组织的文件夹结构 +'logs/{{start_time.format("YYYY")}}/{{start_time.format("MM")}}/{{start_time.format("DD")}}' +// 输出: logs/2021/01/01 + +// 时间范围文件夹 +'data/{{start_time.format("YYYY-MM-DD")}}_to_{{end_time.format("YYYY-MM-DD")}}' +// 输出: data/2021-01-01_to_2021-01-02 +``` + +## 参考文档 + +- [Handlebars 官方文档](https://handlebarsjs.com/guide/) +- [Day.js 官方文档](https://day.js.org/docs/en/display/format) +- [Day.js 格式化选项](https://day.js.org/docs/en/display/format#list-of-all-available-formats) + --- 通过以上详细步骤和说明，管理员可全面、准确地对设备数采客户端进行配置，确保设备数据采集工作的高效运行。 diff --git a/docs/use-case/4-online-simulation-test.md b/docs/use-case/4-online-simulation-test.md index 936e6a59d..ca4790b4a 100644 --- a/docs/use-case/4-online-simulation-test.md +++ b/docs/use-case/4-online-simulation-test.md @@ -92,7 +92,7 @@ tar -czvf install.tar.gz ./install - 源码构建与编译后的可执行程序 -可参考平台文档 👉 https://docs.coscene.cn/docs/sim-and-tests/regression/test-bundle-management +可参考 [Test Bundle 管理文档](../sim-and-tests/regression/4-test-bundle-management.md) 获取更多信息 **_对于企业用户，还支持 GitHub / GitLab 持续集成上传构建产物，跳过手动上传过程。_** @@ -127,7 +127,7 @@ test_case/ ### 文件结构规范 -请严格遵循[平台规定的路径结构](https://docs.coscene.cn/docs/sim-and-tests/regression/intro)： +请严格遵循[平台规定的路径结构](../sim-and-tests/regression/1-intro.md)： ![文件结构示意图](./img/4-online-simulation-test-13.png) @@ -152,7 +152,7 @@ test_case/ - 手动选择记录执行 - 按标签或版本筛选测试集 -详细配置参考 👉 https://docs.coscene.cn/docs/sim-and-tests/regression/config-management +详细配置请参考[测试配置管理文档](../sim-and-tests/regression/3-config-management.md) ![4-online-simulation-test-18](./img/4-online-simulation-test-18.png) ![4-online-simulation-test-19](./img/4-online-simulation-test-19.png) ![4-online-simulation-test-20](./img/4-online-simulation-test-20.png) diff --git a/i18n/en/docusaurus-plugin-content-docs/current/device/4-device-collector.md b/i18n/en/docusaurus-plugin-content-docs/current/device/4-device-collector.md index ab314bc1e..90072013e 100644 --- a/i18n/en/docusaurus-plugin-content-docs/current/device/4-device-collector.md +++ b/i18n/en/docusaurus-plugin-content-docs/current/device/4-device-collector.md @@ -67,7 +67,7 @@ mod: listen_dirs: - /home/bag/ - # When the current time exceeds the file update time by more than {skip_period_hours} hours, the file will not be monitored or collected + # When the current time exceeds the file update time by more than {skip_period_hours} hours, the file will not be monitored skip_period_hours: 2 # Collection directories on the device side, used as specified directories for data collection tasks and rule collection in the project @@ -75,6 +75,14 @@ mod: - /home/bag/ - /home/log/ + # Recursive directory traversal setting - whether to traverse all nested subdirectories for listen_dirs and collect_dirs, defaults to false + recursively_walk_dirs: true + + # Additional files to upload during manual collection tasks - supports both folders and files with absolute paths; supports time variable templates + additional_files: + - /home/just2004noetic/Downloads/testcase + - /additional_files/{{start_time.format('YYYY-MM-DD')}}/log/ + # Assuming the machine has a /home/coscene/device.yaml file with the following content: # soft_version: v1.0 # @@ -117,10 +125,18 @@ Mainly responsible for configuring device-end data storage location information: - `listen_dirs`: Device monitoring directories, used as rule monitoring directories in projects. -- `skip_period_hours`: Files won't be monitored/collected when the time difference between current time and file update time exceeds `{skip_period_hours}`. +- `skip_period_hours`: Files won't be monitored when the time difference between current time and file update time exceeds `{skip_period_hours}`. + +- `recursively_walk_dirs`: Whether to traverse files in nested subdirectories when monitoring and collection directories contain subdirectories. Defaults to false, only reads files in the current directory level. - `collect_dirs`: Device collection directories, used as specified directories for project data collection tasks and rule collection. +- `additional_files`: + - Optional field + - During manual collection tasks, supports collecting additional file information (such as maps, logs, etc.). Supports both folder and file absolute paths. Folders will be recursively traversed for all subfiles. + - Paths support template variables, such as `/home/{{start_time.format('YYYY-MM-DD')}}/log/`. Template variables will be replaced with corresponding collection task time information during collection tasks. Variables will be dynamically updated based on the selected collection start and end times. For specific syntax, please refer to [Template Syntax Usage](#template-syntax-usage) + + ```yaml mod: # mod name, default is 'default', supports monitoring files in specified device directories @@ -149,6 +165,14 @@ mod: collect_dirs: - /home/bag/ - /home/log/ + + # Recursively traverse all subdirectories under /home/bag/ and /home/log/ + recursively_walk_dirs: true + + # For manual collection task starting at 2025-06-27 14:00:00, additionally upload all files from /home/just2004noetic/Downloads/testcase and /home/2025-06-27/log directories, including subdirectories + additional_files: + - /home/just2004noetic/Downloads/testcase + - /home/{{start_time.format('YYYY-MM-DD')}}/log/ ``` ### Device Event Properties (device) @@ -180,6 +204,96 @@ topics: --- +### Template Syntax Usage + +## Overview + +Template syntax is a powerful tool for dynamically generating strings, particularly suitable for dynamically generating file paths and filenames based on time ranges. Based on the [Handlebars](https://handlebarsjs.com/) template engine, integrated with the [Day.js](https://day.js.org/) time processing library. + +By mastering these template syntaxes, you can easily create dynamic file paths and filenames, greatly improving the flexibility and automation of data management. + +## Basic Syntax + +### Supported Variables + +Currently supports the following two time variables: + +- `start_time` - The start time of the current collection task +- `end_time` - The end time of the current collection task + +### Basic Template Format + +``` +{{variable_name.format('format_string')}} +``` + +### Examples + +```javascript +// Basic usage +'{{start_time.format("YYYY-MM-DD")}}' +// Output: "2021-01-01" + +// Combined usage +'data/{{start_time.format("YYYY")}}/{{start_time.format("MM")}}/file.log' +// Output: "data/2021/01/file.log" +``` + +## Time Formatting + +### Common Formatting Options + +Based on [Day.js formatting documentation](https://day.js.org/docs/en/display/format), supports the following formats: + +| Format | Output | Description | +|--------|--------|-------------| +| `YYYY` | 2021 | 4-digit year | +| `YY` | 21 | 2-digit year | +| `MM` | 01-12 | Month (zero-padded) | +| `M` | 1-12 | Month | +| `DD` | 01-31 | Date (zero-padded) | +| `D` | 1-31 | Date | +| `HH` | 00-23 | Hour (zero-padded) | +| `H` | 0-23 | Hour | +| `mm` | 00-59 | Minute (zero-padded) | +| `m` | 0-59 | Minute | +| `ss` | 00-59 | Second (zero-padded) | +| `s` | 0-59 | Second | + +### Formatting Examples + +```javascript +// Date formats +'{{start_time.format("YYYY-MM-DD")}}' // 2021-01-01 +'{{start_time.format("YYYY/MM/DD")}}' // 2021/01/01 +'{{start_time.format("MM-DD-YYYY")}}' // 01-01-2021 + +// Time formats +'{{start_time.format("HH:mm:ss")}}' // 10:30:00 +'{{start_time.format("HH-mm")}}' // 10-30 + +// Combined formats +'{{start_time.format("YYYY-MM-DD HH:mm:ss")}}' // 2021-01-01 10:30:00 +'{{start_time.format("YYYYMMDD_HHmmss")}}' // 20210101_103000 + +// Folder structure organized by year/month/day +'logs/{{start_time.format("YYYY")}}/{{start_time.format("MM")}}/{{start_time.format("DD")}}' +// Output: logs/2021/01/01 + +// Time range folders +'data/{{start_time.format("YYYY-MM-DD")}}_to_{{end_time.format("YYYY-MM-DD")}}' +// Output: data/2021-01-01_to_2021-01-02 +``` + +## Reference Documentation + +- [Handlebars Official Documentation](https://handlebarsjs.com/guide/) +- [Day.js Official Documentation](https://day.js.org/docs/en/display/format) +- [Day.js Formatting Options](https://day.js.org/docs/en/display/format#list-of-all-available-formats) + +--- + Through these detailed steps and instructions, administrators can comprehensively and accurately configure the device data collection client, ensuring efficient operation of device data collection. If you have any questions, please feel free to contact us for support. + diff --git a/i18n/en/docusaurus-plugin-content-docs/current/use-case/4-online-simulation-test.md b/i18n/en/docusaurus-plugin-content-docs/current/use-case/4-online-simulation-test.md new file mode 100644 index 000000000..8bb786863 --- /dev/null +++ b/i18n/en/docusaurus-plugin-content-docs/current/use-case/4-online-simulation-test.md @@ -0,0 +1,198 @@ +--- +sidebar_position: 4 +--- + +# Cloud-Based Simulation & Automated Algorithm Verification + +## Background + +In **robotics R&D**, the endless cycle of testing and validating algorithms and software is often the most time‑consuming and error‑prone part of daily work. + +Sound familiar? + +- You tweak _one_ line of code and end up testing all day +- Your local simulation stack suddenly breaks—re‑installing is a minefield +- Multiple teammates run their own tests … and the results never match + +For teams working on **mobile robots and embodied intelligence**, the pain is even sharper: + +- Rapid software iterations demand continuous validation +- Simulation stacks are complex and costly to maintain +- No unified test standards, so results are hard to quantify +- Limited local compute; long queues for every test run + +--- + +At **coScene** we kept asking ourselves: + +_What if you could validate every algorithm change on demand—zero setup, fully reproducible, and with standardized outputs?_ + +Leveraging our deep experience in **data loops** and **productivity tooling**, we built a one‑stop **cloud simulation platform** grounded in the **SceneOps philosophy**: + +**Just five quick steps to set things up.** +Upload your program and the platform auto‑runs the tests, captures data, and generates visual reports—fully online, zero local dependencies! + +![4-online-simulation-test-1](./img/4-online-simulation-test-1.png) + +This guide walks through Ubuntu 22.04 + ROS 2 Humble + Gazebo as an example. +For other simulators, see the official site: [coScene Platform](https://www.coscene.cn/). + +--- + +## 🧱 Step 1: Build Your Own Test Runtime + +Every team’s tech stack is unique, so coScene lets you create a custom container image with all your dependencies and push it to the platform. + +A minimal `Dockerfile` template: + +```Dockerfile +# syntax=docker/dockerfile:1.2 +ARG BASE_IMAGE=osrf/ros:humble-desktop-full +FROM ${BASE_IMAGE} as base +... +WORKDIR /action/ros2_ws +``` + +Open‑source sample 👉 [Regression_Test-Sample](https://github.com/coscene-io/Regression_Test-Sample/blob/main/Dockerfile) + +After building, push the image to your organization’s private registry so everyone can reuse the same, stable test environment. + +1. Sign in to coScene, open **Organization → Image Registry** to get your push URL + ![4-online-simulation-test-2](./img/4-online-simulation-test-2.PNG) + ![4-online-simulation-test-3](./img/4-online-simulation-test-3.PNG) + ![4-online-simulation-test-4](./img/4-online-simulation-test-4.PNG) + +2. Log in to Docker and push the image + ![4-online-simulation-test-5](./img/4-online-simulation-test-5.PNG) + +3. Verify the image inside the platform and copy its link + ![4-online-simulation-test-6](./img/4-online-simulation-test-6.png) + +You now have a shared “runtime base” for all future simulation tests, with every dependency baked in. + +--- + +## 📦 Step 2: Upload Your Algorithm & Test Program + +After modifying your code, build and package it: + +```bash +tar -czvf install.tar.gz ./install # recommended: package the install/ folder +``` + +In your coScene project, click **Upload Program**. +The platform auto‑extracts the archive and places it in the test container. + +![4-online-simulation-test-8](./img/4-online-simulation-test-8.png) +![4-online-simulation-test-9](./img/4-online-simulation-test-9.png) +![4-online-simulation-test-10](./img/4-online-simulation-test-10.png) +![4-online-simulation-test-11](./img/4-online-simulation-test-11.png) + +Supported uploads: + +- Compiled executables or raw source to be built in‑container + +Refer to the [Test Bundle Management Docs](../sim-and-tests/regression/4-test-bundle-management.md) for more info. + +**Enterprise Edition**: integrate GitHub/GitLab CI to push artifacts automatically—no manual upload required. + +--- + +## 🧪 Step 3: Prepare Test Cases + +Within **Organization → Project → Records** you can define multiple test cases. +Each record is a virtual “test scene” containing: + +- **Map**: `map.pgm` + `map.yaml` +- **Robot model** (e.g., `turtlebot3_waffle_pi`) +- **Config** (e.g., `case.yaml`) +- **Gazebo world** +- Any extra resources + +Example layout: + +``` +test_case/ +├── map.pgm +├── map.yaml +├── case.yaml +├── world.world +└── extra_configs/ +``` + +![4-online-simulation-test-12](./img/4-online-simulation-test-12.png) + +:::info Important Note + +### Directory Convention + +Follow the [official path structure](../sim-and-tests/regression/1-intro.md): + +![Structure diagram](./img/4-online-simulation-test-13.png) + +::: + +### Tagging + +Add labels (e.g., `Navigation_Error`) for easy filtering: + +![Step 1: Tag manager](./img/4-online-simulation-test-14.png) +![Step 2: New tag](./img/4-online-simulation-test-15.png) +![Step 3: Set properties](./img/4-online-simulation-test-16.png) +![Step 4: Apply tag](./img/4-online-simulation-test-17.png) + +You can now select cases by tag in the next step. + +--- + +## ⚙️ Step 4: Configure Test Triggers + +Each project supports flexible trigger rules: + +- Auto‑run tests on every program upload +- Manually select specific records +- Filter by tag or version + +Refer to the [Regression Config Management](../sim-and-tests/regression/3-config-management.md) for more info. + +![4-online-simulation-test-18](./img/4-online-simulation-test-18.png) +![4-online-simulation-test-19](./img/4-online-simulation-test-19.png) +![4-online-simulation-test-20](./img/4-online-simulation-test-20.png) + +--- + +## 📊 Step 5: Run & Monitor Tests + +During execution you get: + +- Live visualization of robot state (Gazebo viewer supported) +- Automatic data recording +- Standardized test reports + +![4-online-simulation-test-21](./img/4-online-simulation-test-21.png) +![4-online-simulation-test-22](./img/4-online-simulation-test-22.png) +![4-online-simulation-test-23](./img/4-online-simulation-test-23.gif) + +Report includes: + +- 📍 Trajectory plots +- ✅ Pass / fail metrics +- 🪵 Logs & exceptions +- 🖼️ Charts & statistics + +![4-online-simulation-test-24](./img/4-online-simulation-test-24.png) +![4-online-simulation-test-25](./img/4-online-simulation-test-25.png) + +--- + +## 🚀 Wrap‑Up: Offload the Grind, Focus on Innovation + +After every code change, you no longer need to: + +- Rebuild environments +- Re‑run scripts by hand +- Manually diff logs + +Just **one upload**, and coScene handles the rest—standardized, automated, and always reproducible—so you can spend more time on what truly matters. + +👉 Sign up and try it out ![https://www.coscene.cn/](https://www.coscene.cn/)