Skip to main content

Intake Agents 1.0

· 3 min read

What's New​

Intake Agents reach their first stable release, marking them as production-ready for data ingestion. They replace Data Pumps as the recommended approach for importing data into DCH.

Key advantages over Data Pumps:

  • Language-agnostic development: Build agents in any programming language using a standardized REST API contract
  • Decoupled architecture: Operate independently of DCH, simplifying updates and maintenance
  • Enhanced performance: Process data in batches instead of loading everything at once, avoiding memory constraints
  • Better observability: Batch processing provides immediate feedback and complete traceability throughout the import process
info

Data Pumps remain supported for backward compatibility, but we strongly recommend migrating to Intake Agents to take advantage of these improvements.

Intake Agent Core​

.NET Development Support: A NuGet package is now available to speed up Intake Agent development in .NET. This open-source package (GPL-3.0 license) provides:

  • Pre-built REST API contract implementation
  • Development utilities to streamline agent creation
  • Built-in Admin UI with extensible custom page support

The package is available on NuGet and serves as the recommended starting point for building Intake Agents in .NET environments. More information can be found in the README inside the package or in Intake Agent Core repository.

Athena Intake Agent​

Athena Intake Agent replaces the Athena Data Pump with enhanced capabilities:

  • Multiple cluster support: Configure connections to different Athena clusters
  • Flexible data retrieval: Query database objects directly or define custom SQL queries
  • Integrated configuration: Create Data Source Configurations directly from your database connections

CSV Intake Agent​

CSV Intake Agent replaces the CSV Data Pump and is additionally open-sourced on GitHub under GPL-3.0 license. Key features include:

  • Local and remote files: Upload from disk or reference server-hosted CSV files
  • Flexible authentication: Support for basic auth, OAuth, and other authentication methods

SQL Intake Agent​

SQL Intake Agent replaces the MSSQL, PostgreSQL, and Oracle Data Pumps with unified capabilities:

  • Multi-database support: Connect to MSSQL, PostgreSQL, and Oracle databases from a single agent
  • Flexible data retrieval: Query database objects directly or define custom SQL queries
  • Integrated configuration: Create Data Source Configurations directly from your database connections

Structured Data Intake Agent​

Structured Data Intake Agent replaces the JSON Data Pump with expanded format support for JSON and XML files:

  • Local and remote files: Upload from disk or reference server-hosted JSON and XML files
  • Flexible authentication: Support for basic auth, OAuth, and other authentication methods
  • Multi-file configurations: Combine multiple files within a single Data Source Configuration
  • REST API integration: Automate operations programmatically via REST API