Connecting to a Remote HTTP API

Conduit supports two API-related connection types:

Webhook Connection: This type is used when an external system sends data directly to Conduit, where it is then stored. See more.
Remote HTTP API: Conduit initiates a request to an external system to retrieve data, but it does not store the data itself. This setup is particularly valuable for application developers who manage data within their own databases.

In this article we are focusing on Remote HTTP API.

How to Configure the Remote HTTP API Connection

Open the Connections screen in Conduit and click Connect.
Begin by entering a descriptive name to easily identify the connection.
You’ll then access a configuration screen to define the connection settings, which include:
- URL: The HTTP endpoint address.
- Request Method: Specify either GET or POST.
- Request Parameters: Customize the data sent with the request (details in the Request Parameters section below).
- Columns: Define the data structure expected from the server response (see Columns section for further details).

Request Parameters

These allow customization of data sent with requests to the external server, with four types of parameters:

QUERY: Added to the URL as ?param1=value1&param2=value2.
HEADER: Passed in the HTTP HEADER.
JSON: Sets Content-Type to application/json and includes all JSON parameters in the request body as a JSON string (incompatible with BODY parameters).
BODY: Encodes parameters in application/x-www-form-urlencoded format (incompatible with JSON parameters).

OAuth

For APIs that requires OAuth, please consider using an alternative approach:
Custom Integration

Request Parameters Placeholders

Conduit often works with time-interval data (e.g., statistics, tickets). To support this, placeholders can be used in request parameters to automatically substitute dates at runtime. Supported placeholders include:

{{days}}: Number of days in a specified period (e.g., 7 for October 15-21).
{{date_from}}: Start date in YYYY-MM-DD format.
{{date_to}}: End date in YYYY-MM-DD format.

API Response Configuration

Currently, only responses in CSV and JSON formats are supported.

CSV:
In the case of a CSV, we expect the server to return a CSV file with the column headers in the first row.

Define columns:

Select Names.
Select Data types.

If the response file includes columns beyond those specified in Columns, they will be ignored. Missing columns in the response that are defined in Columns will result in an error.

JSON:
Remote APIs provide data in JSON format, while Conduit's internal API requires flat table inputs. To convert JSON into a flat table, you must define the process for interpreting and mapping the JSON structure.

Let's consider an example. An HTTP API returns a list of SpaceX rocket launches in JSON format:

[
  {
    "capsule_serial": "C101",
    "capsule_id": "dragon1",
    "status": "retired"
  },
  {
    "capsule_serial": "C102",
    "capsule_id": "dragon1",
    "status": "retired",
  }
]

We need to transform this data into a flat list.

capsule_serial	capsule_id	status
C101	dragon1	retired
C102	dragon1	retired

Define the mapping from JSON to CSV columns in the "mapping" section:

Define:

rows_accessor: iterator on elements in JSON. Please refer to JSONPath.
name: name of a CSV column
accessor: JSONPath of the JSON element.

Please see more details on how to use JSONPath including nested attributes see here.

Limitations

Timeout: 15 seconds to connect, 60 seconds to receive a response.
File Size: Up to 50 MB.

The Remote HTTP API connection is available within the following Conduit product areas:

AI
Workflows
Dashboards

When the "Share to Workspaces" checkbox is selected, the connection becomes available in child workspaces in a read-only mode.

* In order to have this option, please create an Organization Name.

Also, when creating a workspace through the API, the ext_id parameter can optionally be provided (e.g., a user ID from the developer's database).

So during requests, the saved ext_id value is substituted into the {{user_id}} parameter.

See also