Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
115 changes: 113 additions & 2 deletions docs/device/4-device-collector.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,14 +75,22 @@ mod:
listen_dirs:
- /home/bag/

# 当前时间距离文件更新时间超出 {skip_period_hours} 小时的时候,文件不会被监听/采集
# 当前时间距离文件更新时间超出 {skip_period_hours} 小时的时候,文件不会被监听
skip_period_hours: 2

# 设备端的采集目录,作为项目中数据采集任务与规则采集的指定目录
collect_dirs:
- /home/bag/
- /home/log/

# 递归遍历所有子文件夹设置,针对监听目录和采集目录,是否遍历填写目录下的所有嵌套子文件夹,默认为否
recursively_walk_dirs: true

# 手动任务采集时指定上传额外文件,支持文件夹和文件,填写绝对路径;支持特定的时间变量模版
additional_files:
- /home/just2004noetic/Downloads/testcase
- /additional_files/{{start_time.format('YYYY-MM-DD')}}/log/

# 假设机器端存在 /home/coscene/device.yaml 文件,其内容为
# soft_version: v1.0
#
Expand Down Expand Up @@ -124,7 +132,9 @@ collector:
- 选填项
- 对应标识码字段名

- `skip_period_hours`:若当前时间距离文件更新时间超出设定时间,文件不会被监听/采集。
- `skip_period_hours`:若当前时间距离文件更新时间超出设定时间,文件不会被监听。

- `recursively_walk_dirs`:当监听和采集目录存在子文件夹,是否遍历嵌套子文件夹中的文件。默认为否,只读取当前文件夹下的一级所有文件

- `listen_dirs`:
- 选填项
Expand All @@ -133,6 +143,11 @@ collector:

- `collect_dirs`:设备端的采集目录,作为项目中数据采集任务与规则采集的指定目录。

- `additional_files`:
- 选填项
- 手动任务采集时,支持补充采集额外的文件信息(如地图、日志等)。支持填写文件夹以及文件绝对路径,文件夹会递归遍历所有的子文件
- 路径支持模版变量信息,如 `/home/{{start_time.format('YYYY-MM-DD')}}/log/`, 模版变量在采集任务中会替换成对应的采集任务时间信息,依据选择的采集开始和截止时间修改,变量会绑定更新,渲染为对应的值。具体的语法请参考 [模板语法使用](#模板语法使用)

```yaml
mod:
# mod 名称,默认 default,支持监听设备端指定目录下的文件,若有自定义的监听形式,请联系刻行时空
Expand Down Expand Up @@ -160,6 +175,14 @@ mod:
collect_dirs:
- /home/bag/
- /home/log/

# 会递归遍历 /home/bag/ 和 /home/log/ 目录下的所有子文件夹
recursively_walk_dirs: true

# 手动采集任务开始时间为 2025-06-27 14:00:00,额外补充上传 /home/just2004noetic/Downloads/testcase 和 /home/2025-06-27/log 目录下的所有文件,包含子文件夹
additional_files:
- /home/just2004noetic/Downloads/testcase
- /home/{{start_time.format('YYYY-MM-DD')}}/log/
```

### 设备事件属性(device)
Expand Down Expand Up @@ -189,6 +212,94 @@ topics:
- error_code
```

---
### 模板语法使用

## 概述

模板语法是一种动态生成字符串的强大工具,特别适用于根据时间范围动态生成文件路径和文件名。基于 [Handlebars](https://handlebarsjs.com/) 模板引擎,集成了 [Day.js](https://day.js.org/) 时间处理库。

通过掌握这些模板语法,您可以轻松创建动态的文件路径和文件名,大大提高数据管理的灵活性和自动化程度。

## 基本语法

### 支持的变量

目前支持以下两个时间变量:

- `start_time` - 当前采集任务的开始时间
- `end_time` - 当前采集任务的结束时间

### 基本模板格式

```
{{变量名.format('格式字符串')}}
```

### 示例

```javascript
// 基本用法
'{{start_time.format("YYYY-MM-DD")}}'
// 输出: "2021-01-01"

// 组合使用
'data/{{start_time.format("YYYY")}}/{{start_time.format("MM")}}/file.log'
// 输出: "data/2021/01/file.log"
```

## 时间格式化

### 常用格式化选项

基于 [Day.js 格式化文档](https://day.js.org/docs/en/display/format),支持以下格式:

| 格式 | 输出 | 描述 |
|------|------|------|
| `YYYY` | 2021 | 四位年份 |
| `YY` | 21 | 两位年份 |
| `MM` | 01-12 | 月份(补零) |
| `M` | 1-12 | 月份 |
| `DD` | 01-31 | 日期(补零) |
| `D` | 1-31 | 日期 |
| `HH` | 00-23 | 小时(补零) |
| `H` | 0-23 | 小时 |
| `mm` | 00-59 | 分钟(补零) |
| `m` | 0-59 | 分钟 |
| `ss` | 00-59 | 秒(补零) |
| `s` | 0-59 | 秒 |

### 格式化示例

```javascript
// 日期格式
'{{start_time.format("YYYY-MM-DD")}}' // 2021-01-01
'{{start_time.format("YYYY/MM/DD")}}' // 2021/01/01
'{{start_time.format("MM-DD-YYYY")}}' // 01-01-2021

// 时间格式
'{{start_time.format("HH:mm:ss")}}' // 10:30:00
'{{start_time.format("HH-mm")}}' // 10-30

// 组合格式
'{{start_time.format("YYYY-MM-DD HH:mm:ss")}}' // 2021-01-01 10:30:00
'{{start_time.format("YYYYMMDD_HHmmss")}}' // 20210101_103000

// 按年月日组织的文件夹结构
'logs/{{start_time.format("YYYY")}}/{{start_time.format("MM")}}/{{start_time.format("DD")}}'
// 输出: logs/2021/01/01

// 时间范围文件夹
'data/{{start_time.format("YYYY-MM-DD")}}_to_{{end_time.format("YYYY-MM-DD")}}'
// 输出: data/2021-01-01_to_2021-01-02
```

## 参考文档

- [Handlebars 官方文档](https://handlebarsjs.com/guide/)
- [Day.js 官方文档](https://day.js.org/docs/en/display/format)
- [Day.js 格式化选项](https://day.js.org/docs/en/display/format#list-of-all-available-formats)

---

通过以上详细步骤和说明,管理员可全面、准确地对设备数采客户端进行配置,确保设备数据采集工作的高效运行。
Expand Down
6 changes: 3 additions & 3 deletions docs/use-case/4-online-simulation-test.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ tar -czvf install.tar.gz ./install

- 源码构建与编译后的可执行程序

可参考平台文档 👉 https://docs.coscene.cn/docs/sim-and-tests/regression/test-bundle-management
可参考 [Test Bundle 管理文档](../sim-and-tests/regression/4-test-bundle-management.md) 获取更多信息

**_对于企业用户,还支持 GitHub / GitLab 持续集成上传构建产物,跳过手动上传过程。_**

Expand Down Expand Up @@ -127,7 +127,7 @@ test_case/

### 文件结构规范

请严格遵循[平台规定的路径结构](https://docs.coscene.cn/docs/sim-and-tests/regression/intro):
请严格遵循[平台规定的路径结构](../sim-and-tests/regression/1-intro.md):

![文件结构示意图](./img/4-online-simulation-test-13.png)

Expand All @@ -152,7 +152,7 @@ test_case/
- 手动选择记录执行
- 按标签或版本筛选测试集

详细配置参考 👉 https://docs.coscene.cn/docs/sim-and-tests/regression/config-management
详细配置请参考[测试配置管理文档](../sim-and-tests/regression/3-config-management.md)
![4-online-simulation-test-18](./img/4-online-simulation-test-18.png)
![4-online-simulation-test-19](./img/4-online-simulation-test-19.png)
![4-online-simulation-test-20](./img/4-online-simulation-test-20.png)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -67,14 +67,22 @@ mod:
listen_dirs:
- /home/bag/

# When the current time exceeds the file update time by more than {skip_period_hours} hours, the file will not be monitored or collected
# When the current time exceeds the file update time by more than {skip_period_hours} hours, the file will not be monitored
skip_period_hours: 2

# Collection directories on the device side, used as specified directories for data collection tasks and rule collection in the project
collect_dirs:
- /home/bag/
- /home/log/

# Recursive directory traversal setting - whether to traverse all nested subdirectories for listen_dirs and collect_dirs, defaults to false
recursively_walk_dirs: true

# Additional files to upload during manual collection tasks - supports both folders and files with absolute paths; supports time variable templates
additional_files:
- /home/just2004noetic/Downloads/testcase
- /additional_files/{{start_time.format('YYYY-MM-DD')}}/log/

# Assuming the machine has a /home/coscene/device.yaml file with the following content:
# soft_version: v1.0
#
Expand Down Expand Up @@ -117,10 +125,18 @@ Mainly responsible for configuring device-end data storage location information:

- `listen_dirs`: Device monitoring directories, used as rule monitoring directories in projects.

- `skip_period_hours`: Files won't be monitored/collected when the time difference between current time and file update time exceeds `{skip_period_hours}`.
- `skip_period_hours`: Files won't be monitored when the time difference between current time and file update time exceeds `{skip_period_hours}`.

- `recursively_walk_dirs`: Whether to traverse files in nested subdirectories when monitoring and collection directories contain subdirectories. Defaults to false, only reads files in the current directory level.

- `collect_dirs`: Device collection directories, used as specified directories for project data collection tasks and rule collection.

- `additional_files`:
- Optional field
- During manual collection tasks, supports collecting additional file information (such as maps, logs, etc.). Supports both folder and file absolute paths. Folders will be recursively traversed for all subfiles.
- Paths support template variables, such as `/home/{{start_time.format('YYYY-MM-DD')}}/log/`. Template variables will be replaced with corresponding collection task time information during collection tasks. Variables will be dynamically updated based on the selected collection start and end times. For specific syntax, please refer to [Template Syntax Usage](#template-syntax-usage)


```yaml
mod:
# mod name, default is 'default', supports monitoring files in specified device directories
Expand Down Expand Up @@ -149,6 +165,14 @@ mod:
collect_dirs:
- /home/bag/
- /home/log/

# Recursively traverse all subdirectories under /home/bag/ and /home/log/
recursively_walk_dirs: true

# For manual collection task starting at 2025-06-27 14:00:00, additionally upload all files from /home/just2004noetic/Downloads/testcase and /home/2025-06-27/log directories, including subdirectories
additional_files:
- /home/just2004noetic/Downloads/testcase
- /home/{{start_time.format('YYYY-MM-DD')}}/log/
```

### Device Event Properties (device)
Expand Down Expand Up @@ -180,6 +204,96 @@ topics:

---

### Template Syntax Usage

## Overview

Template syntax is a powerful tool for dynamically generating strings, particularly suitable for dynamically generating file paths and filenames based on time ranges. Based on the [Handlebars](https://handlebarsjs.com/) template engine, integrated with the [Day.js](https://day.js.org/) time processing library.

By mastering these template syntaxes, you can easily create dynamic file paths and filenames, greatly improving the flexibility and automation of data management.

## Basic Syntax

### Supported Variables

Currently supports the following two time variables:

- `start_time` - The start time of the current collection task
- `end_time` - The end time of the current collection task

### Basic Template Format

```
{{variable_name.format('format_string')}}
```

### Examples

```javascript
// Basic usage
'{{start_time.format("YYYY-MM-DD")}}'
// Output: "2021-01-01"

// Combined usage
'data/{{start_time.format("YYYY")}}/{{start_time.format("MM")}}/file.log'
// Output: "data/2021/01/file.log"
```

## Time Formatting

### Common Formatting Options

Based on [Day.js formatting documentation](https://day.js.org/docs/en/display/format), supports the following formats:

| Format | Output | Description |
|--------|--------|-------------|
| `YYYY` | 2021 | 4-digit year |
| `YY` | 21 | 2-digit year |
| `MM` | 01-12 | Month (zero-padded) |
| `M` | 1-12 | Month |
| `DD` | 01-31 | Date (zero-padded) |
| `D` | 1-31 | Date |
| `HH` | 00-23 | Hour (zero-padded) |
| `H` | 0-23 | Hour |
| `mm` | 00-59 | Minute (zero-padded) |
| `m` | 0-59 | Minute |
| `ss` | 00-59 | Second (zero-padded) |
| `s` | 0-59 | Second |

### Formatting Examples

```javascript
// Date formats
'{{start_time.format("YYYY-MM-DD")}}' // 2021-01-01
'{{start_time.format("YYYY/MM/DD")}}' // 2021/01/01
'{{start_time.format("MM-DD-YYYY")}}' // 01-01-2021

// Time formats
'{{start_time.format("HH:mm:ss")}}' // 10:30:00
'{{start_time.format("HH-mm")}}' // 10-30

// Combined formats
'{{start_time.format("YYYY-MM-DD HH:mm:ss")}}' // 2021-01-01 10:30:00
'{{start_time.format("YYYYMMDD_HHmmss")}}' // 20210101_103000

// Folder structure organized by year/month/day
'logs/{{start_time.format("YYYY")}}/{{start_time.format("MM")}}/{{start_time.format("DD")}}'
// Output: logs/2021/01/01

// Time range folders
'data/{{start_time.format("YYYY-MM-DD")}}_to_{{end_time.format("YYYY-MM-DD")}}'
// Output: data/2021-01-01_to_2021-01-02
```

## Reference Documentation

- [Handlebars Official Documentation](https://handlebarsjs.com/guide/)
- [Day.js Official Documentation](https://day.js.org/docs/en/display/format)
- [Day.js Formatting Options](https://day.js.org/docs/en/display/format#list-of-all-available-formats)

---

Through these detailed steps and instructions, administrators can comprehensively and accurately configure the device data collection client, ensuring efficient operation of device data collection.

If you have any questions, please feel free to contact us for support.

Loading