cartodb4/cartodb
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Initial commit

Browse Source
This commit is contained in:
zhongjin
2020-06-15 10:58:47 +08:00
commit 4f1dfe7564
8590 changed files with 1516878 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/developer-center/import-api/guides/01-quickstart.md
+81
View File
@@ -0,0 +1,81 @@
## Quickstart
For this example (and the rest of the ones illustrated here) we will be using a command-line tool known as `cURL`. For more info about this tool see [this blog post](http://quickleft.com/blog/command-line-tutorials-curl) or type `man curl` in bash.
### Uploading a Local File
Suppose you have a CARTO account whose username is *documentation*, and you want to upload a local file named *prism_tour.csv* (located in the *Documents* folder). This requires that you execute the following command on a Terminal window:
#### Call
```bash
curl -v -F file=@/home/documentation/Documents/prism_tour.csv
"https://documentation.carto.com/api/v1/imports/?api_key=3102343c42da0f1ffe6014594acea8b1c4e7fd64"
```
Note that the *api_key* element has an alphanumeric value that is exclusive to the *documentation* CARTO account.
The response to this request appears in the following format, where a successful value indicates that the import process is enqueued:
#### Response
```
{
"item_queue_id": "efa9925c-31dd-11e4-a95e-0edbca4b5057",
"success": true
}
```
The `item_queue_id` value is a unique identifier that references the import process. Once this process has started, its information can be obtained doing a request to the imports endpoint as explained in the ["Check the status of an import process]({{site.importapi_docs}}/guides/standard-tables/#check-the-status-of-an-import-process) section.
### Uploading from a Remote URL
Suppose you have a server at the hostname *examplehost.com*, with a csv named *sample.csv* already uploaded. Creating a table from the URL requires that you execute the following command on a Terminal window:
#### Call
```bash
curl -v -H "Content-Type: application/json" -d '{"url":"https://examplehost.com/sample.csv"}'
"https://documentation.carto.com/api/v1/imports/?api_key=3102343c42da0f1ffe6014594acea8b1c4e7fd64"
```
The response to this request returns the following format, returning a success value if the import process is correctly enqueued:
#### Response
```
{
"item_queue_id": "efa9925c-31dd-11e4-a95e-0edbca4b5057",
"success": true
}
```
### Connecting to a Database
Suppose you have an external MySQL database named _mydb_ that you want to connect to. For the purpose of this example, you will access a server with the address of _mydbserver.com_. Your username is _myuser_, and your password is _mypass_. Connect a CARTO dataset to a remote table, named _mytable_, by executing the following command on a Terminal window:
#### Call
```bash
curl -v -H "Content-Type: application/json" -d '{
"connector": {
"provider": "mysql",
"connection": {
"server":"mydatabaserver.com",
"database":"mydb",
"username":"myuser,
"password":"mypass"
},
"table": "mytable"
}
}' "https://documentation.carto.com/api/v1/imports/?api_key=3102343c42da0f1ffe6014594acea8b1c4e7fd64"
```
#### Response
```
{
"item_queue_id": "tyf9925c-32dd-11f4-a95f-0fdbca4b5058",
"success": true
}
```
doc/developer-center/import-api/guides/02-general-concepts.md
+33
View File
@@ -0,0 +1,33 @@
## General Concepts
The following concepts are the same for every endpoint in the API except when it is noted explicitly.
### Auth
Manipulating data on a CARTO account requires prior authentication using a unique identifier as a password. For the import API, a special identifier known as the API Key is used as a proof of authentication for each user account to authorize access to its data.
To execute an authorized request, `api_key=YOURAPIKEY` should be added to the request URL. This parameter can be also passed as a POST parameter. We **strongly advise** using HTTPS when you are performing requests that include your `api_key`.
---
### Errors
Errors are reported using standard HTTP codes and extended information encoded in the HTML language, as shown in the following example:
```html
<html>
<head>
<title>411 Length Required</title>
</head>
<body bgcolor="white">
<center>
<h1>411 Length Required</h1>
</center>
<hr>
<center>nginx</center>
</body>
</html>
```
Depending on the specific case, additional information regarding the error may be presented. See support section for details about known error codes and solutions.
doc/developer-center/import-api/guides/03-standard-tables.md
+212
View File
@@ -0,0 +1,212 @@
## Standard Tables
A standard import stores the data you upload from files with valid formats, creating tables at CARTO. These are the default tables used to store the data of the uploaded files (that will be used to create datasets and maps). Any CARTO user may create, manipulate, and delete their own datasets.
### Upload file
##### Definition
```bash
POST api/v1/imports
```
##### Params
Param | Description
--- | ---
api_key | The target CARTO account API key.
file | When importing local files, you need to perform a POST with a file (see the call with cURL example in this section).
url | When importing remote files, the full URL to the publicly accessible file.
type_guessing | If set to `false` disables field type guessing (for Excel and CSVs). Optional. Default is `true`.
quoted_fields_guessing | If set to `false` disables type guessing of CSV fields that come inside double quotes. Optional. Default is `true`.
content_guessing | Set to `true` to enable content guessing and automatic geocoding based on results. Currently, this only implements geocoding of countries, cities and IP addresses. Optional. Default is `false`.
create_vis | Set to `true` to flag the import so that when it finishes, it creates a Map automatically after importing the Dataset. Optional. Default is `false`.
collision_strategy | Determines the behavior when importing a dataset that has the same name as an existing table. By default, it is imported and renamed with a sequential number (`mytable`, `mytable_1`...). Optional. If you set `collision_strategy=skip`, the table with the matching name will not be imported. If you set `collision_strategy=overwrite`, it will replace the table with the matching name, but only if the schemas are compatible.
privacy | Used to set the privacy settings of the table or tables resulting from the import. If **create_vis** is set to true, the resulting visualization privacy settings will also be determined by this parameter. **privacy** can be set to:
--- | ---
<i class="Icon Icon--s5 Icon--cGrey Icon--mAlign Icon--indent"></i> public | The resulting table or visualization can be viewed by anyone.
<i class="Icon Icon--s5 Icon--cGrey Icon--mAlign Icon--indent"></i> private | The resulting table or visualization can only be viewed by the uploader.
<i class="Icon Icon--s5 Icon--cGrey Icon--mAlign Icon--indent"></i> link | The resulting table or visualization can only be viewed through a private link shared by the uploader.
table_name | Used to duplicate one of your existing tables. **Do not mix with File/URL imports**.
table_copy | Similar to *table_name*, internally used for table copying. **Do not set**.
table_id | Internal usage for table migrations. **Do not set**.
append | Reserved for future usage. **Do not set**.
sql | Used to create a new table from a SQL query applied to one of your tables. **Do not mix with File/URL imports**.
service_name | Used to upload from datasources, indicates which datasource to use. Check [here](https://github.com/CartoDB/cartodb/tree/master/services/datasources/lib/datasources) for an updated list of available datasources to use. **Intended for CARTO Builder usage**.
service_item_id | Used to upload from datasources and indicates data of the datasource. Check [here](https://github.com/CartoDB/cartodb/tree/master/services/datasources/lib/datasources) for an updated list of available datasources to use. **Intended for CARTO Builder usage**.
##### Response
The response includes:
Attributes | Description
--- | ---
item_queue_id | A unique alphanumeric identifier referencing the import process in the targeted account.
success | A boolean value indicating whether the import process was started or not.
#### Local File Upload Example
##### Call
```bash
curl -v -F file=@/path/to/local/file "https://{account}.carto.com/api/v1/imports/?api_key={account API Key}"
```
##### Response
```
{
"item_queue_id": "9906bce0-f1a3-4b07-be71-818f4bfd7673",
"success": true
}
```
#### URL Upload Example
##### Call
```bash
curl -v -H "Content-Type: application/json" -d '{"url":"https://remotehost.url/path/to/remotefile"}'
"https://{account}.carto.com/api/v1/imports/?api_key={account API Key}"
```
##### Response
```
{
"item_queue_id": "9906bce0-f1a3-4b07-be71-818f4bfd7673",
"success": true
}
```
### Check the status of an import process
When uploading a file for import, it may take some time due to the file's size and the additional processing on the CARTO side. Using this request, an import process state can be retrieved.
##### Definition
```bash
GET /api/v1/imports/<import_id>
```
##### Params
Param | Description
--- | ---
api_key | The target CARTO account API key.
The import identifer | A unique alphanumeric element that identifies the import process to be retrieved. It is the *item_queue_id* element returned after running the upload request successfully.
##### Response
The response includes the following items:
Attributes | Description
--- | ---
id | A unique identifier for the import process. It is the same as the *import id* provided in the request.
user_id | A unique alphanumeric element that identifies the CARTO account user in the internal database.
table_id | A unique alphanumeric element that identifies the created table in the internal CARTO database.
data_type | This element identifies the service type used to import the file. Possible values are `file`, `url`, `external_table`, `query`, `table` or `datasource`.
table_name | The final name of the created table in the targeted CARTO account. It usually has the same name as the uploaded file, unless there already exists a table with the same name (in this case, an integer number is appended to the table name).
state | A string value indicating the current state of the importing process. It can have any of the following values: *enqueued*, *pending*, *uploading*, *unpacking*, *importing*, *guessing*, *complete*, or *failure*.
error_code | A number corresponding to the error code in case of failure during the import process, that is, when the *success* item has a *false* value.
queue_id | A unique identifier for the import process in the importing queue. It is the same as the *import id* provided in the request.
tables_created_count | The number of tables that the import process generated. For multi-file uploads, this value can be greater than one. If the import process fails, its value will be `null`.
synchronization_id | This element has a *null* value when the import is not configured as a Sync Table.
type_guessing | A boolean indicating whether field type guessing (for Excel and CSVs) is enabled or not.
quoted_fields_guessing | A boolean indicating whether type guessing of CSV fields inside double quotes is enabled for the data import.
content_guessing | A boolean indicating whether content guessing and automatic geocoding is enabled for the data import.
create_visualization | A boolean indicating whether the import process will create a map automatically or not. Its value corresponds to the import option `create_vis` chosen by the user.
visualization_id | A unique identifier for the map created in the import process. Only applies if `create_visualization` is set to `true`.
user_defined_limits | Internal usage for user limits.
get_error_text | This element contains an error description to be outputted in case of a failure during the import process. It contains the error title and description, its source (`user` or `cartodb`), and troubleshooting details.
display_name | Similar to `table_name`. For `url` uploads, it shows the name of the file. Otherwise, it shows the import id.
success | A boolean value indicating whether the import process succeeded (*true* or *false*).
warnings | A text field containing warning messages related to the import process, if applicable.
is_raster | A boolean value indicating whether the imported table contains raster data or not.
#### Example
##### Call
```bash
curl -v "https://{account}.carto.com/api/v1/imports/{import_id}?api_key={account API Key}
```
##### Response
```
{
id: "029a6053-b2fb-43dd-baa6-805d679c404f",
user_id: "ca8c5ace-d573-450b-8a43-6c7eafadd80e",
table_id: null,
data_type: "url",
table_name: null,
state: "failure",
error_code: 1002,
queue_id: "029a6053-b2fb-43dd-baa6-805d679c404f",
tables_created_count: null,
synchronization_id: null,
type_guessing: true,
quoted_fields_guessing: true,
content_guessing: false,
create_visualization: false,
visualization_id: null,
user_defined_limits: "{"twitter_credits_limit":0}",
get_error_text: {
title: "Unsupported/Unrecognized file type",
what_about: "Should we support this filetype? Let us know in our <a href='mailto:support@carto.com'>support email</a>!",
source: "user"
},
display_name: "shapefile_streets.cpg",
success: false,
warnings: null,
is_raster: false
}
```
---
### Retrieving a list of all the current import processes
Lists the import identifiers of the files that are being imported in the targeted CARTO account.
##### Definition
```bash
GET /api/v1/imports/
```
##### Params
Param | Description
--- | ---
api_key | The target CARTO account API key.
##### Response
The response includes:
Attributes | Description
--- | ---
imports | A list of unique alphanumeric identifiers referencing the import processes in the targeted CARTO account.
success | A boolean value indicating if the request was successful.
#### Example
##### Call
```bash
curl -v "https://{account}.carto.com/api/v1/imports/?api_key={account API Key}"
```
##### Response
```
{
"imports": [
"1234abcd-1234-1a2b-3c4d-4321dcba5678"
],
"success": true
}
```
doc/developer-center/import-api/guides/04-map-visualizations.md
+74
View File
@@ -0,0 +1,74 @@
## CARTO Map Visualizations
The *Export map* option enables you to download a map, and the connected dataset, as a .carto file. This is useful for downloading complete CARTO visualizations that you can share or import.
**Note:** The Import API export visualization command only works for maps created from your dashboard.
A cURL POST request allows you to export the visualization, which you will have to poll with a GET command until the state is `complete`.
### Export a CARTO Visualization
#### Call
```bash
curl -H 'Content-Type: application/json' https://{username}.carto.com/api/v3/visualization_exports\?api_key\={api_key} -X POST --data '{"visualization_id":"{visualization_id}"}'
```
##### Params
Param | Description
--- | ---
api_key | The target CARTO account API key.
visualization_id | A unique identifier for the map created in the export process. Only applies if `create_visualization` is set to true when the map was created.
#### Response
```
{"id":"b94c26f3-fa16-4f13-b672-45ebbd5a9c95","visualization_id":"ace62506-brc8-6570-2p91-8vf3af3ftc44","user_id":"42b78090-6a11-475a-8060-0a90322752af2","state":"pending","url":null,"created_at":"2016-05-05T09:36:09+00:00","updated_at":"2016-05-05T09:36:09+00:00"}
```
_After making the POST request to create the export, it is expected that the request will take some time. You must poll the server by making a GET request, until state becomes complete._ For example:
```
curl -v -H 'Content-Type: application/json' https://{username}.carto.com/api/v3/visualization_exports/{visualization_export_id}\?api_key\={api_key} -X GET
```
Once completed, the response status changes to `complete` and displays the upload url location of the .carto visualization file:
```
{"id":"b94c26f3-fa16-4f13-b672-45ebbd5a9c95","visualization_id":"{visualization_id}","user_id":"42b78090-6a11-475a-8060-0a90322752af2","state":"complete","url":"/uploads/6a2b6fbd86e2c750160a/ace62506-brc8-6570-2p91-8vf3af3ftc44.carto","created_at":"2016-05-05T09:36:09+00:00","updated_at":"2016-05-05T09:36:13+00:00"}%
```
The response includes:
Attributes | Description
--- | ---
id | A unique identifier for the export process. It is the same as the _export id_ provided in the request.
vizualization_id | A unique identifier for the map created in the export process. Only applies if `create_visualization` is set to true when the map was created.
user_id | A unique alphanumeric element that identifies the CARTO account user in the internal database.
state | A string value indicating the current state of the export process. It can have any of the following values: _enqueued, pending, uploading, unpacking, importing, guessing, complete_, or _failure_.
url | The **public** URL address where the file to be exported is located.
created_at | The date time at which the visualization was created in the CARTO database.
updated_at | The date time at which the visualization had its contents modified.
##### Example
Example POST request:
```
curl -v -H 'Content-Type: application/json' https://{username}.carto.com/api/v3/visualization_exports\?api_key\={api_key} -X POST --data '{"visualization_id":"9a0f4384-afe3-412a-8b09-136b7d9a4013"}'
{"id":"72e488a6-cf0e-404d-bc7e-de9c9840aadf","visualization_id":"9a0f4384-afe3-412a-8b09-136b7d9a4013","user_id":"d80fe0f5-0465-4c4a-a2fe-1f14a93f3c5b","state":"pending","url":null,"created_at":"2016-05-20T07:01:38+00:00","updated_at":"2016-05-20T07:01:38+00:00"}%
```
Example completed response:
```
curl -H 'Content-Type: application/json' https://{username}.carto.com/api/v3/visualization_exports/72e488a6-cf0e-404d-bc7e-de9c9840aadf\?api_key\=04039a13c1bdda65df8bd825b3b8e8117444c950 -X GET
{"id":"72e488a6-cf0e-404d-bc7e-de9c9840aadf","visualization_id":"9a0f4384-afe3-412a-8b09-136b7d9a4013","user_id":"d80fe0f5-0465-4c4a-a2fe-1f14a93f3c5b","state":"complete","url":"http://s3.amazonaws.com/com.cartodb.imports.production/9d149e72331bf074e35f/9a0f4384-afe3-412a-8b09-136b7d9a4013.carto?AWSAccessKeyId=AKIAJK5S64CVBE35QTKA&Expires=1463734900&Signature=b4cwrdoB%2B0FlTelIzNOAgslUDXY%3D","created_at":"2016-05-20T07:01:38+00:00","updated_at":"2016-05-20T07:01:40+00:00"}%
```
### Import a CARTO Visualization
To import a .carto visualization, you can use the standard Import API procedure for uploading a local file.
See the import errors list on Support section to troubleshoot any importing errors.
doc/developer-center/import-api/guides/05-sync-tables.md
+311
View File
@@ -0,0 +1,311 @@
## Sync Tables
Sync tables are available for certain CARTO plans. These tables store data from a remote file and refresh their contents during periodic intervals specified by the user. The base files from which the sync tables retrieve their contents may come from Google Drive, Dropbox, Box or a public URL.
### Upload and manage synced tables
##### Definition
```html
GET /api/v1/synchronizations
```
##### Params
Param | Description
--- | ---
api_key | The target CARTO account API key.
##### Response
The response includes an **array** of items, each one containing the following elements:
Attributes | Description
--- | ---
id | A unique alphanumeric identifier of the synced table.
name | The actual name of the created sync table.
interval | An integer value representing the number of seconds between synchronizations of the table contents.
url | The **public** URL address where the file to be synchronized is located.
state | A string value indicating the current state of the synchronized dataset. It can have any of the following values: **created**, **queued**, **syncing**, **success** or **failure**.
created_at | The date time at which the table was created in the CARTO database.
updated_at | The date time at which the table had its contents modified.
run_at | The date time at which the table will get its contents synched with the source file.
retried_times | An integer value indicating the number of attempts that were performed to sync the table.
log_id | A unique alphanumeric identifier to locate the log traces of the given table.
error_code | An integer value representing a unique error identifier.
error_message | A string value indicating the message related to the *error_code* element.
ran_at | The date time at which the table **had** its contents synched with the source file.
modified_at | The date time at which the table was manually modified, if applicable.
etag | HTTP entity tag of the source file.
checksum | See **etag**.
user_id | A unique alphanumeric element that identifies the CARTO account user in the internal database.
service_name | A string with the name of the datasource used to import the file. It can have any of the following values: *gdrive* for Google Drive, *dropbox* for Dropbox and *null* for URL imports.
service_item_id | A unique identifier used by CARTO to reference the sync table and its related datasource service.
type_guessing | A boolean indicating whether field type guessing (for Excel and CSVs) is enabled or not.
quoted_fields_guessing | A boolean indicating whether type guessing of CSV fields inside double quotes is enabled for the data import.
content_guessing | A boolean indicating whether content guessing and automatic geocoding is enabled for the data import.
visualization_id | A unique identifier for the map created in the import process. Only applies if `create_visualization` is set to true.
from_external_source | A boolean indicating whether the Sync Table is connected to an external source, generally the CARTO Data library.
Finally, the array includes a **total_entries** element that indicates the number of items contained in the response array.
#### Example
##### Call
```bash
curl -v "https://{username}.carto.com/api/v1/synchronizations/?api_key={account API Key}"
```
##### Response
```javascript
{
"synchronizations": [
{
"id": "246dae5e-5302-1ae5-af51-0e85ad047bba",
"name": "barrios_5",
"interval": 2592000,
"url": "https://common-data.carto.com/api/v2/sql?q=select+*+from+%22barrios%22&format=shp&filename=barrios",
"state": "success",
"created_at": "2015-09-04T12:40:37+00:00",
"updated_at": "2016-02-01T12:45:07+00:00",
"run_at": "2016-03-02T12:45:07+00:00",
"retried_times": 0,
"log_id": "2d9b4a52-1daa-429b-b425-6ad561609cb1",
"error_code": null,
"error_message": null,
"ran_at": "2016-02-01T12:45:07+00:00",
"modified_at": "2015-04-22T12:17:50+00:00",
"etag": null,
"checksum": null,
"user_id": "cf8a5cce-d573-4a0b-8c43-6caeaf1dd80e",
"service_name": null,
"service_item_id": "https://common-data.carto.com/api/v2/sql?q=select+*+from+%22barrios%22&format=shp&filename=barrios",
"type_guessing": true,
"quoted_fields_guessing": true,
"content_guessing": true,
"visualization_id": "2954fa60-5a02-11e5-888a-0e5e07bb5d8a",
"from_external_source": false
}
],
"total_entries": 1
}
```
---
### Syncing a file from a URL
##### Definition
```html
POST /api/v1/synchronizations
```
##### Params
Param | Description
--- | ---
api_key | The targeted CARTO account API key.
url | The **public** URL address where the file to be imported is located.
interval | The number of seconds for the synchronization period. *Note*: Sync interval must be at least 900 (15 minutes).
type_guessing | If set to *false* disables field type guessing (for Excel and CSVs). Optional. Default is *true*.
quoted_fields_guessing | If set to *false* disables type guessing of CSV fields that come inside double quotes. Optional. Default is *true*.
content_guessing | Set it to *true* to enable content guessing and automatic geocoding based on results. Currently it only implements geocoding of countries. Optional. Default is *false*.
##### Response
The response includes the following items:
Attributes | Description
--- | ---
endpoint | This item refers to the internal CARTO controller code responsible for performing the import.
item_queue_id | A unique alphanumeric identifier that refers to the import process. It can be used to retrieve data related to the created table.
id | An alphanumeric identifier used internally by CARTO as a reference to the import process.
name | This item is currently deprecated.
interval | An integer value that stores the number of seconds between synchronizations.
state | A string value indicating the current condition of the importing process. It can have any of the following values: **created**, **queued**, **syncing**, **success** or **failure**.
user_id | A unique alphanumeric identifier to reference the user in the CARTO Engine.
created_at | The date time at which the table was created in the CARTO Engine.
updated_at | The date time at which the table had its contents modified.
run_at | The date time at which the table will get its contents synched with the source file.
ran_at | The date time at which the table **had** its contents synched with the source file.
modified_at | The date time at which the table was manually modified, if applicable.
etag | HTTP entity tag of the source file.
checksum | See **etag**.
log_id | A unique alphanumeric identifier to locate the log traces of the given table.
error_code | An integer value representing a unique error identifier.
error_message | A string value indicating the message related to the *error_code* element.
retried_times | An integer value indicating the number of attempts that were performed to sync the table.
service_name | A string with the name of the datasource used to import the file. It can have any of the following values: *gdrive* for Google Drive, *dropbox* for Dropbox and *null* for URL imports.
service_item_id | A unique identifier used by CARTO to reference the sync table and its related datasource service.
type_guessing | A boolean indicating whether field type guessing (for Excel and CSVs) is enabled or not.
quoted_fields_guessing | A boolean indicating whether type guessing of CSV fields inside double quotes is enabled for the data import.
content_guessing | A boolean indicating whether content guessing and automatic geocoding is enabled for the data import.
visualization_id | A unique identifier for the map created in the import process. Only applies if `create_visualization` is set to true.
from_external_source | A boolean indicating whether the Sync Table is connected to an external source, generally the CARTO Common-Data library.
#### Example
##### Call
```bash
curl -v -H "Content-Type: application/json" -d '{"url":"https://public.url.to.file/sample_file", "interval":"3600"}' "https://{username}.carto.com/api/v1/synchronizations/?api_key={account API Key}"
```
##### Response
```
{
"data_import": {
"endpoint": "/api/v1/imports",
"item_queue_id": "1234abcd-1234-1a2b-3c4d-4321dcba5678"
},
"id": "abcd1234-a5b6-c7d8-1a2b-efgh5678abcd",
"name": null,
"interval": 3600,
"url": "https://public.url.to.file/sample_file",
"state": "created",
"user_id": "aaaabbbb-1234-5678-dcba-abcd1234efgh",
"created_at": "2014-08-05T13:39:15+00:00",
"updated_at": "2014-08-05T13:39:15+00:00",
"run_at": "2014-08-05T14:39:15+00:00",
"ran_at": "2014-08-05T13:39:15+00:00",
"modified_at": null,
"etag": null,
"checksum": "",
"log_id": "06fafab8-3502-11e4-9514-0e230854a1cb",
"error_code": null,
"error_message": null,
"retried_times": 0,
"service_name": null,
"service_item_id": null,
"type_guessing": true,
"quoted_fields_guessing": true,
"content_guessing": false,
"visualization_id": null,
"from_external_source": false
}
```
---
### Removing the synchronization feature from a given dataset
A sync table can be converted to a standard dataset (a dataset that never gets synced).
##### Definition
```bash
DELETE /api/v1/synchronizations/<import_id>
```
##### Params
Param | Description
--- | ---
api_key | The target CARTO account API key.
Target table import id | The unique alphanumeric identifier of the target sync dataset.
#### Example
##### Call
```bash
curl -v -X "DELETE" https://{username}.carto.com/api/v1/synchronizations/{import_id}?api_key={account API Key}"
```
##### Response
An HTTP No content - 204 response should result as a confirmation for the removal of the synchronization feature for the target table.
---
### Check whether a sync table is syncing or not
A large synced table may take some time to get fully synced. In the meantime, it could be useful to check whether it finished refreshing its contents.
##### Definition
```bash
GET /api/v1/synchronizations/<import_id>/sync_now
```
##### Params
Param | Description
--- | ---
api_key | The target CARTO account API key.
Target table import id | The unique alphanumeric identifier of the target sync table.
##### Response
The response includes the following items:
Attributes | Description
--- | ---
state | A string value indicating the status of the synchronization. It can have any of the following values: **created**, **queued**, **syncing**, **success** or **failure**.
#### Example
##### Call
```bash
curl -v -X "GET" "https://{username}.carto.com/api/v1/synchronizations/{import_id}/sync_now?api_key={account API Key}"
```
##### Response
```
{
"state": "syncing"
}
```
---
### Force a synchronization action on a sync table
Sync tables have their contents synchronized with the source file in periodic time intervals as specified by the user during the creation process. However, a dataset can be synchronized at an arbitrary moment in time if desired. **Note**: Forcing a synchronization can only be performed when the last synchronization attempt occurred at least 900 seconds (15 minutes) before.
#### Definition
```bash
PUT /api/v1/synchronizations/<import_id>/sync_now
```
##### Params
Param | Description
--- | ---
api_key | The target CARTO account API key.
Target table import id | The unique alphanumeric identifier of the target sync table.
#### Response
The response includes the following items:
Attributes | Description
--- | ---
enqueued | A boolean value indicating whether the request has been successfully appended to the processing queue.
synchronization_id | A unique alphanumeric identifier referring to the queue element just added.
#### Example
##### Call
```bash
curl -v -X "PUT" "https://{username}.carto.com/api/v1/synchronizations/<import_id>/sync_now?api_key={account API Key}" -H "Content-Length:0"
```
##### Response
```bash
{
"enqueued": true,
"synchronization_id": "1234abcd-aaaa-2222-4444-dcba4321a1b2"
}
```
doc/developer-center/import-api/guides/06-the-arcgis-connector.md
+221
View File
@@ -0,0 +1,221 @@
## Importing an ArcGIS&trade; Layer
The ArcGIS&trade; Connector allows you to import ArcGIS&trade; layers into a CARTO account as dataset from ArcGIS Server&trade; (version 10.1 or higher is required). Note that **this connector is disabled by default** in the CARTO importer options. If you are interested in enabling it, please contact [support@carto.com](mailto:support@carto.com) for more details.
**Tip:** You can easily import ArcGIS&trade; server table URLs from CARTO Builder, with the ArcGIS Server&trade; Connect Dataset option.
### Import an ArcGIS&trade; Layer
ArcGIS&trade; layers stored in ArcGIS Server&trade; can get imported as CARTO datasets. Such layers must be (PUBLIC) and accessible via an **ArcGIS&trade; API REST URL**, using the following structure:
```html
http://<host>/<site>/rest/services/<folder>/<serviceName>/<serviceType>/<layer_ID>
```
##### Definition
```bash
POST api/v1/imports
```
##### Params
Param | Description
--- | ---
interval | This value **MUST** be set to *0*. **Different values do not guarantee correct imports**.
service_item_id | The ArcGIS&trade; API REST URL where the ArcGIS&trade; layer is located.
service_name | This value **MUST** be set to *arcgis* to make use of this connector.
value | Same URL as specified in the *service_item_id* parameter.
##### Response
The response includes:
Attributes | Description
--- | ---
item_queue_id | A unique alphanumeric identifier referencing the imported file in the targeted account.
success | A boolean value indicating whether the import process was successfully appended to the processing queue or not.
#### Example
##### Call
```bash
curl -v -H "Content-Type: application/json" -d '{"interval":"0","service_item_id": "http://url.to.arcgis.server.layer", "service_name": "arcgis", "value": "http://url.to.arcgis.server.layer"}' "https://{username}.carto.com/api/v1/imports/?api_key={API_KEY}"
```
##### Response
```
{
"item_queue_id": "d676fd50-b774-4052-a4f1-e56ac6a4300e",
"success": true
}
```
### Syncing an ArcGIS&trade; Layer
An ArcGIS&trade; layer can get imported to a CARTO account as a synchronized table. The target ArcGIS&trade; layer must be (PUBLIC) and accessible via an ArcGIS&trade; API REST URL, using the following structure:
```html
http://<host>/<site>/rest/services/<folder>/<serviceName>/<serviceType>/<layer_ID>
```
##### Definition
```bash
POST /api/v1/synchronizations
```
##### Params
Param | Description
--- | ---
interval | The number of seconds for the synchronization period. *Note*: Sync interval must be at least 900 (15 minutes).
service_item_id | The ArcGIS&trade; API REST URL where the ArcGIS&trade; dataset is located. **Note:** Layers and Datasets must be (PUBLIC) and accessible via an ArcGIS&trade; API REST URL. You cannot enforce ArcGIS Server security parameters into the request, the REST endpoints must be publicly available.
service_name | This value **MUST** be set to *arcgis* to make use of this connector.
url | This value **MUST** be empty.
##### Response
The response includes the following items:
Attributes | Description
--- | ---
endpoint | This item refers to the internal CARTO controller code responsible for performing the import.<br/><br/>
item_queue_id | A unique alphanumeric identifier that refers to the import process. It can be used to retrieve data related to the created dataset.
id | An alphanumeric identifier used internally by CARTO as a reference to the import process.
name | This item is currently deprecated.
interval | An integer value that stores the number of seconds between synchronizations.
url | This value is empty in this case.
state | A string value indicating the current condition of the importing process. It can have any of the following values: **created**, **queued**, **syncing**, **success** or **failure**.
user_id | A unique alphanumeric identifier to reference the user in the CARTO Engine.
created_at | The date time at which the dataset was created in the CARTO Engine.
updated_at | The date time at which the dataset had its contents modified.
run_at | The date time at which the table will get its contents synced with the source file.
ran_at | The date time at which the table **had** its contents synced with the source file.
modified_at | The date time at which the dataset was manually modified, if applicable.
etag | HTTP entity tag of the source file.
checksum | See **etag**.
log_id | A unique alphanumeric identifier to locate the log traces of the given dataset.
error_code | An integer value representing a unique error identifier.
error_message | A string value indicating the message related to the *error_code* element.
retried_times | An integer value indicating the number of attempts that were performed to sync the table.
service_name | This value is set to *arcgis*.
service_item_id | This item contains the ArcGIS&trade; API REST URL targeting the imported ArcGIS&trade; layer.
type_guessing | A boolean indicating whether field type guessing (for Excel and CSVs) is enabled or not.
quoted_fields_guessing | A boolean indicating whether type guessing of CSV fields inside double quotes is enabled for the data import.
content_guessing | A boolean indicating whether content guessing and automatic geocoding is enabled for the data import.
visualization_id | A unique identifier for the map created in the import process. Only applies if `create_visualization` is set to true.
from_external_source | A boolean indicating whether the Sync Table is connected to an external source, generally the CARTO Common-Data library.
#### Example
##### Call
```bash
curl -v -H "Content-Type: application/json" -d '{"interval":"604800","service_item_id": "http://url.to.arcgis.server.layer", "service_name": "arcgis", "url":""}' "https://{username}.carto.com/api/v1/synchronizations?api_key={API_KEY}"
```
##### Response
```
{
"endpoint":"/api/v1/imports",
"item_queue_id":"4ff4abdd-9d37-4b7a-8e13-fb00376e2a58",
"id":"d4bc05e8-5063-11e4-9886-0e018d66dc29",
"name":null,
"interval":604800,
"url":"",
"state":"created",
"user_id":"4884b545-07f4-4ce4-a62f-fe9e2412098f",
"created_at":"2014-10-10T09:57:22+00:00",
"updated_at":"2014-10-10T09:57:22+00:00",
"run_at":"2014-10-17T09:57:22+00:00",
"ran_at":"2014-10-10T09:57:22+00:00",
"modified_at":null,
"etag":null,
"checksum":"",
"log_id":"6aa19bf6-42db-477a-9b69-2c4f74fd8c31",
"error_code":null,
"error_message":null,
"retried_times":0,
"service_name":"arcgis",
"service_item_id":"http://url.to.arcgis.layer",
"type_guessing": true,
"quoted_fields_guessing": true,
"content_guessing": true,
"visualization_id": "2954fa60-5a02-11e5-888a-0e5e07bb5d8a",
"from_external_source": false
}
```
### Import an ArcGIS&trade; Dataset
This option allows you to programmatically import a complete set of layers belonging to an ArcGIS&trade; dataset (as opposed to using CARTO Builder ArcGIS Server&trade; Connect Dataset option). Such a dataset must be (PUBLIC) and accessible via an ArcGIS&trade; API REST URL, using the following structure:
```html
http://<host>/<site>/rest/services/<folder>/<serviceName>/<serviceType>/
```
##### Definition
```bash
POST api/v1/imports
```
##### Params
Param | Description
--- | ---
interval | This value **MUST** be set to *0*. **Different values do not guarantee correct imports**.
service_item_id | The ArcGIS&trade; API REST URL where the ArcGIS&trade; dataset is located.
service_name | This value **MUST** be set to *arcgis* to make use of this connector.
value | Same URL as specified in the *service_item_id* parameter
##### Response
The response includes:
Attributes | Description
--- | ---
item_queue_id | A unique alphanumeric identifier referencing the imported file in the targeted account.
success | A boolean value indicating whether the file import succeeded or not.
#### Example
##### Call
```bash
curl -v -H "Content-Type: application/json" -d '{"interval":"0","service_item_id": "http://url.to.arcgis.server.dataset", "service_name": "arcgis", "value": "http://url.to.arcgis.server.dataset"}' "https://{username}.carto.com/api/v1/imports/?api_key={API_KEY}"
```
##### Response
```
{
"item_queue_id": "c478fd50-f984-4091-d1f2-e72ac6c4333e",
"success": true
}
```
### Limits
Connections to ArcGIS&trade; server are limited by two types of timeouts: connection and response timeouts.
#### Connection timeout
Connection timeout to the ArcGIS&trade; server is set to 60 seconds.
This means if the ArcGIS&trade; server does not respond to a request in 60 seconds the connection is closed and the synchronization of the dataset will fail.
In the vast majority of cases a connection timeout means there's something wrong in the ArcGIS&trade; server, so you should contact the server administrator for more details about the issue.
#### Response timeout
Response timeout from the ArcGIS&trade; server is set to 60 seconds.
This means the ArcGIS&trade; server did not finish the request in 60 seconds, after the connection was made, so the resulting dataset will be incomplete and the import will fail with a `Download timeout` error code.
Response timeouts can happen for a number of reasons, the more commons ones are because the ArcGIS&trade; server is overloaded and is not able to respond in a timely manner or the dataset is too big to be transferred in 60 seconds from the server to CARTO.
In any case, we recommend you to first check any issue with the ArcGIS&trade; server administrator and if that does not solve the issue contact us at [support@carto.com](mailto:support@carto.com) for more details.
doc/developer-center/import-api/guides/07-database-connectors.md
+1084
View File
File diff suppressed because it is too large Load Diff
doc/developer-center/import-api/guides/08-importing-geospatial-data.md
+199
View File
@@ -0,0 +1,199 @@
## Importing Geospatial Data
This section explains how importing a dataset creates columns in CARTO (and the naming conventions that you should use). It includes how CARTO guesses content during the import process, lists the supported geospatial formats for uploading data, and describes how to upload multilayer datasets or batch file uploads.
### Dataset Basics
When a file is imported, it is transformed into a dataset that can be processed by CARTO. The system automatically creates the following columns:
* **cartodb_id**
* This column is used as the primary key of the table
* Its values must be integers, non-null, and unique
* **the_geom**
* This column stores the main geometric features of a dataset in the EPSG 4326 projection
* **the_geom_webmercator**
* This column stores the geometries transformed into the EPSG 3857 projection, and is used for rendering purposes
* **_feature_count**
* This column is automatically created when overview representations of data are created (for datasets containing more than 500,000 points).
When a dataset is exported from CARTO, it includes the `cartodb_id` and `the_geom` columns, which will be reused if the dataset is then imported to the system. This ensures that importing an exported dataset contains the original exported dataset content and row order.
If these columns are generated by the user, the [CARTO table requirements](https://github.com/CartoDB/cartodb-postgresql/blob/master/doc/CartoDB-user-table.rst) must be followed in order to produce a successful import. Otherwise, importing datasets which do not meet the requirements (such as a dataset with duplicated integers in its `cartodb_id` column) will result in an import failure.
#### Naming
Apply the following naming conventions for datasets in CARTO, and avoid using the reserved words as part of your file names.
* Table names must begin with a letter (a-z). Otherwise, "table_" is prepended to the name
* Column names must begin with a letter (a-z) or an underscore (_)
* Column and table names can have a maximum of 63 characters. Names are trimmed if they exceed this length
##### Reserved Words
There are certain words reserved in the system that cannot be used to name columns or datasets, mainly the PostgreSQL reserved words. Any names that conflict with a reserved word are prefixed with an underscore (_) automatically.
### Import Guessing
CARTO includes guessing functionality during the import process. This is useful for when files or data are missing some upload information. The following guessing options are available:
* **Fields guessing**
For files whose format does not include type information (usually CSV files), field guessing options can be enabled. There are two particular guessing options for these type of files:
* **Type guessing**: determines the type of imported columns from the text contents, available in the CSV file. If enabled, it generates numeric and boolean columns when appropriate, otherwise, it uses regular string columns
* **Quoted fields guessing**: when disabled, avoids double quoted fields for type guessing. Otherwise, double quoted fields are used when enabled
* **Content Guessing**
Files that contain country, city, IP address information can be automatically geocoded by the system, if the content guessing option is enabled. This automatic geocoding only occurs if there is not a big proportion of repeated, or null values, in a column. Content guessing does not require the target columns to be named in a special way (such as "country" or "city"), CARTO inspects the different available columns and identifies which of them can be guessed geospatially.
**Tip:** For information about how to granularly configure the guessing options for your import process, view the upload file parameters on the standard tables section.
### Supported Geospatial Data Formats
CARTO supports several geospatial data formats to upload vector data. The important details of each format, as well as some guidelines to upload your files to CARTO, are defined in this section.
#### Shapefile
The Shapefile format is a multi-file format — it consists of a set of files with the same name and stored in the same directory, which are differentiated by their extension.
A Shapefile has to be formed, at least, by a .shp file, a .shx file, a .prj file and a .dbf file. These files contain the geometry data, the indexes, the projection information and the attributes, respectively. Other auxiliary files are not mandatory and contain extra information for the Shapefile. Shapefiles must be imported as a single compressed file, in the .zip or .gz format.
**Note:** The Shapefile format has certain limitations that can affect the way that your datasets are exported/imported into CARTO:
* The column name cannot exceed 10 characters. Exporting a dataset with longer names in this format will trim the names
* Date columns only support the date, not the time. Exporting and importing a date column as a Shapefile will remove all time information and maintain just the date. If you need to work with date and time data, it is recommended to export/import the information as a string and convert it to a date
* Although the projection of the file should be correctly determined and adjusted from the .prj file, it is recommended to upload Shapefiles in the EPSG 4326 projection
* For improved compatibility, ensure you save your Shapefile with encoding UTF-8, prior importing
#### Keyhole Markup Language (KML)
The KML format is a XML based format which adds to it a geographical meaning by being able to define features such as points, polygons or lines in the EPSG 4326 projection.
KML uses common XML types such as string, boolean, double, or int, so your column types will be respected when your dataset is imported or exported from CARTO.
Each feature is defined as a Placemark element, which usually contains a name, a description, and the geometry itself. If more data columns are required, these fields need to be defined and included inside a ExtendedData element of the KML document.
In terms of geometric elements, the Point, Polygon, Line, MultiGeometry and Geometry elements are supported. Different geometry types in the same layer are not supported.
#### KMZ
A Keyhole Markup language Zipped (KMZ) file corresponds to a compressed file, including a KML file and zero, or more, supporting files (images, icons, overlays or other elements referenced in the KML file). See the [Keyhole Markup Language (KML) section for more information](#keyhole-markup-language-kml).
#### GeoJSON
The GeoJSON format is an extension of the JavaScript Object Notation (JSON) that encodes geographical features and their metadata. This format supports data types such as string, double or boolean. Dates exported as GeoJSON are stored as strings and will be recognized as such, on data imports.
With respect to geometries, Points, (Multi)Polygons and (Multi)Lines are supported. GeometryCollection geometric objects are not supported and will raise an import error. The supported geometries can be imported inside FeatureCollection and Feature objects.
Importing different geometry types in a FeatureCollection element is not supported.
#### CSV
Comma-Separated Values (or TSV, Tab-Separated Values) files can be imported to CARTO. For a successful import, follow these formatting guidelines:
* The first line of the CSV file must contain the name of the columns
* The rest of the lines of the CSV file must follow the schema defined by the header column, in terms of number of columns
* To ensure correct parsing, it is recommended that string values are double-quoted
* If the data itself contains quotes, the values must be double-quoted and the internal quotes must be escaped
* CSV lines must be terminated with CR/LF, or LF line terminators. CR line terminators are not supported
###### Example: Quoted strings in a CSV
`````
name, description, score
"John Doe", "Awesome, the best player ever", 100
`````
###### Example: Escaped quotes in a CSV
````
name, geojson
"Null Island", "{""type"": ""Point"", ""coordinates"": [0,0]}"
````
##### CSV Format Guessing
As the CSV format does not specify the type of the columns in the data, CARTO applies a guessing functionality that converts your data to columns, using a supported format. This enables you to generate numeric columns, or geocode your dataset directly on import.
There are two particular guessing options for CSV files: types guessing and quoted fields guessing. View the [Import Guessing](#import-guessing) section for details.
#### Spreadsheets (Excel or OpenDocument)
Excel files, or other spreadsheets (such as OpenDocument spreadsheets or Google Drive spreadsheets) are supported by CARTO.
The format of the uploaded Spreadsheet must apply the following format:
* The first row must contain the names for each column
* Merged cells are not supported
* Graphs, charts, or other kind of elements are not supported
For multi-sheet spreadsheets, only the first sheet will be imported. For the case of Google Drive the maximum size of the spreadhseet is limited to 10MB.
#### GPX
The GPX (GPS Exchange Format) files are XML documents that contain waypoints, tracks and/or routes. When importing a GPX file, CARTO will generate different datasets for points, tracks and waypoints. The resulting names of these datasets will be a combination of the GPX name and their type: `_track_points`, `_tracks`, and `_waypoints`, respectively.
#### OSM
CARTO supports importing Open Street Map dumps (.osm files). These files are XML documents that have a `osm` parent element that can contain blocks of nodes, ways, or relations representing points, lines or polygons. CARTO will automatically separate OSM dumps into different tables, depending on the geometry. Therefore, importing a single OSM file can lead to more than one resulting dataset.
#### MapInfo
The MapInfo file format is geospatial vector data developed by MapInfo, which supports grids based multiple files. MapInfo files (.DAT, .ID, .MAP, .TAB) must be imported as a single compressed file, in the .zip or .gz format.
##### CARTO
CARTO files are CARTO generated map visualization files. This .carto file includes the dataset and visualization definition, which contains any SQL queries, CartoCSS, basemaps, attributions, metadata, and styling that was applied to a map. This is useful for downloading complete CARTO visualizations that you can share or import.
#### GeoPackage
GeoPackage (GPKG) files are an [open standard format](https://www.geopackage.org/) for spatial data. The format supports multiple layers, and all the geometry types used by CARTO: Points, (Multi)Polygons and and (Multi)Lines. GeoPackage files can be imported as an uncompressed .GPKG file or as a compressed .ZIP file. Each layer (up to 50) in the GeoPackage file will be imported as a separate CARTO table.
### FileGeodatabase
File Geodatabase (GDB) is a proprietary [Esri format](http://desktop.arcgis.com/en/arcmap/10.3/manage-data/administer-file-gdbs/file-geodatabases.htm) for spatial data. The GDB format is a directory with a `.gdb` extension containing the data files, so for download and upload a zip file containing the directory is used, either with a `.zip` or a `.gdb.zip` extension. Each layer (up to 50) in the GDB file will be imported as a separate CARTO table.
**Note:** The "[personal geodatabase](http://desktop.arcgis.com/en/arcmap/latest/manage-data/administer-file-gdbs/personal-geodatabases.htm)" (having a `.mdb` extension) format used by ArcGIS 8 and ArcGIS 9 is not supported by CARTO.
### Multilayer Uploads
Several of the formats supported by CARTO can store different layers, or geometric types, by definition. Importing a file that contains more than one layer result in different imported datasets.
If the option `create_vis` is enabled in the import process, the different layers imported will be added to the created map. The number of layers that can be included in a map depends on the maximum value of layers per map in the configuration of the user.
The maximum number of datasets created from a multilayer file is 10. If the imported file contains more than 10 layers, those layers are omitted.
**Important note:** The "[personal geodatabase](http://desktop.arcgis.com/en/arcmap/latest/manage-data/administer-file-gdbs/personal-geodatabases.htm)" (having a `.mdb` extension) format used by ArcGIS 8 and ArcGIS 9 is not supported by CARTO.
#### Shapefile
The different layers included in a Shapefile are imported as independent datasets.
#### KML Files
KML files generate a different dataset, per each Folder, that they contain.
#### GPX Files
GPX files that contain more than one type of elements (waypoints, tracks, and/or routes) are imported in a different dataset, per type.
#### OSM Files
OSM files generate a different layer, per each type of geometry that their nodes, ways, or relations represent (points, polygons or lines).
#### GeoPackage
GPKG files generate a different dataset for each layer in the file, up to 50.
#### File Geodatabase
GDB files generate a different dataset for each layer in the file, up to 50.
### Multiple File Uploads
You can perform a batch file upload if the files are sent to the server in a compressed format. As with the case of multilayer uploads, if the import process is configured to generate a map after import, the different datasets are added as layers to the new map. The number of layers that can be included in a map depends on the maximum value of layers allotted to the users account.
The maximum number of files that can be imported in a single file is 10. If the compressed file contains more than 10 files, only the first 10 files are imported and the rest of the files are omitted.
doc/developer-center/import-api/guides/09-column-names-normalization.md
+44
View File
@@ -0,0 +1,44 @@
## Column names normalization
When data is uploaded into CARTO using the Import API, the resulting dataset in CARTO might have different column names than the original dataset.
This is because as part of the import workflow there's a process to normalize the column names to avoid unsupported column names in the datasets created after the import process finishes. Some of the actions taken when normalizing column names are:
- Remove accents
- Convert multiple consecutive underscores and hyphens to a single underscore
- Remove unsupported characters ([]{}&%$+, etc.)
- Remove white spaces
- Avoid duplicated column names
- Avoid column names longer than 63 characters
- Avoid reserved words
- Force lower case names
- Force column names to start by a character
### Some examples of column name normalization
Find below a table with some examples of column names and how they are normalized by the Import API:
| *original column name* | *normalized column name* |
| Field: 2 | field_
| 2 Items | _2_item
| Unnamed: 0 | unnamed_0
| 201moore | _201moore
| 201moore | _201moore_1
| Acadia 1.2.3 | acadia_1_2_3
| _testingTesting | _testingtesting
| 1 | _1
| 1.0 | _1_0
| SELECT | _select
| à | a
| longcolumnshouldbesplittedsomehowanditellyouwhereitsgonnabesplittedrightnow | longcolumnshouldbesplittedsomehowanditellyouwhereitsgonnabespli
| longcolumnshouldbesplittedsomehowanditellyouwhereitsgonnabesplittedrightnow | longcolumnshouldbesplittedsomehowanditellyouwhereitsgonnabe_1
| all | _all
### Changing column names
In some cases you may want to preserve some of the column names in your original data. For example, let's say you have a column name `column__1` which indeed is a valid column name for PostgreSQL and the Import API renamed it as `column_1`. In the case you want to preserve `column__1` as the column name, right after the import process finished you can update the column name using the [CARTO SQL API](https://carto.com/developers/sql-api/) with a query like this:
```sql
ALTER TABLE test_1 RENAME COLUMN column_1 to column__1;
SELECT CDB_TableMetadataTouch('test_1');
```