Upgrade data collection from stitched scripts into platform capability
The unified collection framework connects file, database and telemetry data sources through a plugin model, with built-in lifecycle hooks, resumable transfer, multi-protocol delivery and external-system integration — turning ad-hoc collection scripts into a long-lived, maintainable and observable data-ingress platform.
Core capabilities
File collection plugin
Supports file-event listening, directory traversal, suffix matching and regex filtering — for logs, batch files and landed-data processing.
- File-event listening
- Directory traversal
- Suffix matching
- Event / name regex filtering
Database plugin
Unified access, incremental collection and periodic sync for structured data sources, fitting both business and analytics databases.
- Incremental pull
- Task scheduling
- Unified plugin model
Telemetry plugin
Collects and normalizes data from telemetry, SNMP, NetFlow, sFlow and similar protocols.
- SNMP
- NetFlow
- sFlow
- Extensible telemetry protocols
Reliable transfer
tusd-style resumable transfer for large files and long-haul tasks over unstable networks.
- Resume from breakpoint
- Chunked transfer
- Large-file friendly
- Retry mechanism
Solutions and integration
Lifecycle hooks and transfer-cycle hooks
Trigger hooks at the collection, pre-processing, transfer, callback and post-processing stages, wiring business logic, audit, notification and orchestration into one flow.
- File-event stage hooks
- Transfer-cycle hooks
- Async or blocking modes
- Easy business post-processing
Multi-protocol transfer and external integration
Supports S3, SFTP, HTTP API and other transfer protocols, and invokes hook functions via n8n, MQ, CLI, HTTP API and RPC for open integration.
- S3 / SFTP / HTTP API
- n8n orchestration
- MQ / CLI / RPC
- Open system integration
A data ingress for UAV and BIM scenarios
Data collection is not a standalone product but the unified ingress through which UAV field-flight data, images and video, building models, device telemetry and business records enter the AI platform — connecting collection naturally to downstream recognition, modeling, analysis and operations.
- Field image & video backhaul
- BIM models & business records
- Device telemetry & sensor data
- Standardized input for recognition & workflows
Architecture capabilities
Unified plugin model with multi-source access
File, database and telemetry plugins share one extension model, reducing integration complexity.
- Unified interface
- Unified task scheduling
- Unified observability
A hook mechanism for the collection lifecycle
Connects collection, pre-processing, transfer, callback and business orchestration into one.
- Async execution
- Blocking execution
- Business callbacks
- Exception handling
A reliability-oriented transfer engine
Supports resumable transfer, retries, protocol adaptation and large-file transfer control for complex production environments.
- Resumable transfer
- Retry mechanism
- Multi-protocol adaptation
- State tracking