Initial commit

doc/manual/source/components.rst

@@ -0,0 +1,14 @@
+Components
+==========
+
+A common CARTO stack is powered by those components:
+
+.. toctree::
+
+  components/postgresql.rst
+  components/redis.rst
+  components/cartodb-postgresql.rst
+  components/cartodb-editor.rst
+  components/maps-api.rst
+  components/sql-api.rst
+

doc/manual/source/components/cartodb-editor.rst

@@ -0,0 +1,27 @@
+CARTO Builder
+=============
+
+Builder is the web management component of CARTO. Within Builder you can find all the features available in CARTO. These are some of the most significant tasks you can do with Builder:
+
+  - User management. Credentials, authorization, personal info and billing.
+  - Connect datasets to your CARTO account either by importing your datasets or other ones publicly available.
+  - Create maps from your datasets
+  - Publising and permissions management of datasets and maps
+  - Synchronized tables management
+
+InternallyBuilder is the operations core of CARTO. It manages PostgreSQL metadata database, keep some metadata in sync with Redis, manages new datasets import queues with resque, etc..
+
+It is developed in Ruby on Rails and like the other components of CARTO is Open Source and you can find the source code at `CartoDB/cartodb <https://github.com/CartoDB/cartodb>`_
+
+You can find usage documentation at https://docs.carto.com/cartodb-editor.html
+
+Although you can chechout any branch of the repository most of them are usually work in progress that is not guaranteed to work. In order to run a production ready Editor service you need to use **master** branch.
+
+Service modes
+-------------
+
+The code of CARTO Builder needs to run in two different modes. HTTP server mode and background jobs mode.
+
+The **HTTP server** processes the http requests sent to the service and returns a response synchronously. Any ruby rack server can be used to start the Builder in this mode. Some examples of rack servers are mongrel, webrick, thin or unicorn.
+
+The **background jobs** mode is started with `resque <https://github.com/resque/resque>`_. In this mode the service keep polling some redis keys in order to find pending background jobs. When it finds one, it processes it and change the state of the job in redis. CARTO uses this mode for different type of jobs like datasets imports or synchronized tables.

doc/manual/source/components/cartodb-postgresql.rst

@@ -0,0 +1,15 @@
+CARTO PostgreSQL extension
+--------------------------
+CARTO's PostgreSQL extension can be found at https://github.com/CartoDB/cartodb-postgresql.
+
+This extensions is required by all the components of CARTO and it must be installed in the server where user databases are stored.
+
+It provides functions and other helpers needed by Builder, Maps and SQL APIs like:
+
+  - CartoDBfying functions which convert raw PostgreSQL tables in tables recognized by CARTO by adding some additional columns and triggers
+  - Multiuser schema handling functions
+  - Quota helpers
+  - Cache helpers
+  - etc..
+
+The CARTO extension depends on :ref:`postgis_label`.

doc/manual/source/components/maps-api.rst

@@ -0,0 +1,10 @@
+Maps API
+========
+
+The Maps API provides a node.js based API that allows you to generate maps based on data hosted in your CARTO account by applying custom SQL and CartoCSS to the data
+
+Like the other components of CARTO is Open Source and you can find the source code at `CartoDB/Windshaft-cartodb <https://github.com/CartoDB/windshaft-cartodb>`_
+
+You can find usage documentation at https://docs.carto.com/cartodb-platform/maps-api.html
+
+Although you can chechout any branch of the repository most of them are usually work in progress that is not guaranteed to work. In order to run a production ready Maps API service you need to use **master** branch.

doc/manual/source/components/postgresql.rst

@@ -0,0 +1,25 @@
+PostgreSQL
+----------
+
+`PostgreSQL <http://postgresql.org>`_ is the open source database powering CARTO.
+
+CARTO uses PostgreSQL for two purposes:
+
+* **Metadata storage**. This is the metadata used by the CARTO Builder. Builder models like users information, visualizations, or other metadata is stored in this database. The name and connection information of this database is specified on the Rails ``app_config.yml`` configuration file.
+
+* **User data storage**. Each single user or organization in CARTO has a individual PostgreSQL database. This database is created during the user signup process. Its database name and connection info is generated on the fly by the CARTO application during this process. Both values are stored within the user info in the metadata database. Every user database name contains the user UUID.
+
+Both metadata database and users databases can be hosted either in the same PostgreSQL cluster or different ones. Having both the in the same cluster is the recommended approach for small environments.
+Builder only knows how to connect to metadata database. However, within every request it checks the connection info of the user database which is stored in metadata database, as described before.
+
+At this moment CARTO requires PostgreSQL 10 version.
+
+.. _postgis_label:
+
+PostGIS
+-------
+`PostGIS <http://postgis.net>`_ is the extension which adds spatial capabilities to PostgreSQL.
+It allows working with geospatial types or running geospatial functions in PostgreSQL.
+Is is required both on the user data clusters and the metadata cluster.
+
+At this moment CARTO requires PostGIS version 2.4.

doc/manual/source/components/redis.rst

@@ -0,0 +1,21 @@
+Redis
+-----
+
+Redis is a key-value store engine used by most components of the CARTO application stack to store configuration and cache.
+
+In contrast to the PostgreSQL metadata (which is used only by the CARTO Builder), the metadata stored in Redis is shared among Builder, the SQL API, and the Maps API.
+
+.. important::
+  Even though also used as a cache, there is also persistent data stored in Redis.
+  In some environments Redis is configured by default to act without persistency
+  (see http://redis.io/topics/persistence).
+
+  You must ensure Redis is configured properly to keep its data between restarts.
+
+Data is stored in separate databases inside this Redis:
+ - Database 0: Table and visualization metadata, including map styles and named maps.
+ - Database 3: OAuth credentials metadata.
+ - Database 5: Metadata about the users, including API keys and database_hosts.
+
+
+At this moment CARTO requires Redis version 4.0 or newer with the `redis-cell` extension.

doc/manual/source/components/sql-api.rst

@@ -0,0 +1,10 @@
+SQL API
+=======
+
+The SQL API provides a node.js based API for running SQL queries against CARTO.
+
+Like the other components of CARTO is Open Source and you can find the source code at `CartoDB/CartoDB-SQL-API <http://github.com/cartodb/cartodb-sql-api>`_
+
+You can find usage documentation at https://docs.carto.com/cartodb-platform/sql-api.html.
+
+Although you can chechout any branch of the repository most of them are usually work in progress that is not guaranteed to work. In order to run a production ready SQL API service you need to use **master** branch.

doc/manual/source/conf.py

@@ -0,0 +1,269 @@
+# -*- coding: utf-8 -*-
+#
+# CartoDB Platform Documentation documentation build configuration file, created by
+# sphinx-quickstart on Wed Jun 24 10:50:07 2015.
+#
+# This file is execfile()d with the current directory set to its
+# containing dir.
+#
+# Note that not all possible configuration values are present in this
+# autogenerated file.
+#
+# All configuration values have a default; values that are commented out
+# serve to show the default.
+
+import sys
+import os
+import sphinx_rtd_theme
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#sys.path.insert(0, os.path.abspath('.'))
+
+# -- General configuration ------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+#needs_sphinx = '1.0'
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    'sphinx.ext.autodoc',
+    'sphinx.ext.doctest',
+    'sphinx.ext.intersphinx',
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# The suffix of source filenames.
+source_suffix = '.rst'
+
+# The encoding of source files.
+#source_encoding = 'utf-8-sig'
+
+# The master toctree document.
+master_doc = 'index'
+
+# General information about the project.
+project = u'CartoDB Platform Documentation'
+copyright = u'2015, CartoDB, Inc.'
+
+# The version info for the project you're documenting, acts as replacement for
+# |version| and |release|, also used in various other places throughout the
+# built documents.
+#
+# The short X.Y version.
+version = '1.0.0'
+# The full version, including alpha/beta/rc tags.
+release = '1.0.0'
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#language = None
+
+# There are two options for replacing |today|: either, you set today to some
+# non-false value, then it is used:
+#today = ''
+# Else, today_fmt is used as the format for a strftime call.
+#today_fmt = '%B %d, %Y'
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+exclude_patterns = []
+
+# The reST default role (used for this markup: `text`) to use for all
+# documents.
+#default_role = None
+
+# If true, '()' will be appended to :func: etc. cross-reference text.
+#add_function_parentheses = True
+
+# If true, the current module name will be prepended to all description
+# unit titles (such as .. function::).
+#add_module_names = True
+
+# If true, sectionauthor and moduleauthor directives will be shown in the
+# output. They are ignored by default.
+#show_authors = False
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = 'sphinx'
+
+# A list of ignored prefixes for module index sorting.
+#modindex_common_prefix = []
+
+# If true, keep warnings as "system message" paragraphs in the built documents.
+#keep_warnings = False
+
+
+# -- Options for HTML output ----------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+html_theme = 'sphinx_rtd_theme'
+html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
+
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further.  For a list of options available for each theme, see the
+# documentation.
+#html_theme_options = {}
+
+# Add any paths that contain custom themes here, relative to this directory.
+#html_theme_path = []
+
+# The name for this set of Sphinx documents.  If None, it defaults to
+# "<project> v<release> documentation".
+#html_title = None
+
+# A shorter title for the navigation bar.  Default is the same as html_title.
+#html_short_title = None
+
+# The name of an image file (relative to this directory) to place at the top
+# of the sidebar.
+#html_logo = None
+
+# The name of an image file (within the static path) to use as favicon of the
+# docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
+# pixels large.
+#html_favicon = None
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
+
+# Add any extra paths that contain custom files (such as robots.txt or
+# .htaccess) here, relative to this directory. These files are copied
+# directly to the root of the documentation.
+#html_extra_path = []
+
+# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
+# using the given strftime format.
+#html_last_updated_fmt = '%b %d, %Y'
+
+# If true, SmartyPants will be used to convert quotes and dashes to
+# typographically correct entities.
+#html_use_smartypants = True
+
+# Custom sidebar templates, maps document names to template names.
+#html_sidebars = {}
+
+# Additional templates that should be rendered to pages, maps page names to
+# template names.
+#html_additional_pages = {}
+
+# If false, no module index is generated.
+#html_domain_indices = True
+
+# If false, no index is generated.
+#html_use_index = True
+
+# If true, the index is split into individual pages for each letter.
+#html_split_index = False
+
+# If true, links to the reST sources are added to the pages.
+#html_show_sourcelink = True
+
+# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
+#html_show_sphinx = True
+
+# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
+#html_show_copyright = True
+
+# If true, an OpenSearch description file will be output, and all pages will
+# contain a <link> tag referring to it.  The value of this option must be the
+# base URL from which the finished HTML is served.
+#html_use_opensearch = ''
+
+# This is the file name suffix for HTML files (e.g. ".xhtml").
+#html_file_suffix = None
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = 'CartoDBPlatformDocumentationdoc'
+
+
+# -- Options for LaTeX output ---------------------------------------------
+
+latex_elements = {
+# The paper size ('letterpaper' or 'a4paper').
+#'papersize': 'letterpaper',
+
+# The font size ('10pt', '11pt' or '12pt').
+#'pointsize': '10pt',
+
+# Additional stuff for the LaTeX preamble.
+#'preamble': '',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title,
+#  author, documentclass [howto, manual, or own class]).
+latex_documents = [
+  ('index', 'CartoDBPlatformDocumentation.tex', u'CartoDB Platform Documentation Documentation',
+   u'CartoDB, Inc.', 'manual'),
+]
+
+# The name of an image file (relative to this directory) to place at the top of
+# the title page.
+#latex_logo = None
+
+# For "manual" documents, if this is true, then toplevel headings are parts,
+# not chapters.
+#latex_use_parts = False
+
+# If true, show page references after internal links.
+#latex_show_pagerefs = False
+
+# If true, show URL addresses after external links.
+#latex_show_urls = False
+
+# Documents to append as an appendix to all manuals.
+#latex_appendices = []
+
+# If false, no module index is generated.
+#latex_domain_indices = True
+
+
+# -- Options for manual page output ---------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+    ('index', 'cartodbplatformdocumentation', u'CartoDB Platform Documentation',
+     [u'CartoDB, Inc.'], 1)
+]
+
+# If true, show URL addresses after external links.
+#man_show_urls = False
+
+
+# -- Options for Texinfo output -------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+#  dir menu entry, description, category)
+texinfo_documents = [
+  ('index', 'CartoDBPlatformDocumentation', u'CartoDB Platform Documentation',
+   u'CartoDB, Inc.', 'CartoDBPlatformDocumentation', 'One line description of project.',
+   'Miscellaneous'),
+]
+
+# Documents to append as an appendix to all manuals.
+#texinfo_appendices = []
+
+# If false, no module index is generated.
+#texinfo_domain_indices = True
+
+# How to display URL addresses: 'footnote', 'no', or 'inline'.
+#texinfo_show_urls = 'footnote'
+
+# If true, do not generate a @detailmenu in the "Top" node's menu.
+#texinfo_no_detailmenu = False
+
+
+# Example configuration for intersphinx: refer to the Python standard library.
+intersphinx_mapping = {'http://docs.python.org/': None}

doc/manual/source/configuration.rst

@@ -0,0 +1,199 @@
+Configuration
+=============
+
+In this section you can find some helpful configuration examples related with **Basemaps**,
+**Domainles Urls** and **Common-data**.
+
+Basemaps
+--------
+
+The way to add/change the basemaps available in CARTO is chaging the
+config/app_config.yml. Basically you need to add a new entry called basemaps,
+that entry can have different sections and each section one or more basemaps.
+
+Each section corresponds to row in CARTO basemap dialog. If the basemaps entry
+is not present a set of default basemaps will be used (CARTO and Stamen ones,
+check the default basemaps file
+https://github.com/CartoDB/cartodb/blob/master/lib/assets/javascripts/cartodb/table/default_layers.js)
+
+Also, it's always necessary to have a default basemap among all the confifured
+ones in the app_config.yml. The way to set a basemap as default a "default"
+attribute needs to be added to the basemap. There can be several basemaps in the
+config with the attribute default set, however, only the first one found in the
+same order than in the app_config will be used as default.
+
+Here is an example config.yml:
+
+.. code-block:: yaml
+
+  basemaps:
+      CARTO:
+        positron_rainbow:
+          default: true # Ident with spaces not with tab
+          url: 'http://{s}.basemaps.cartocdn.com/light_all/{z}/{x}/{y}.png'
+          subdomains: 'abcd'
+          minZoom: '0'
+          maxZoom: '18'
+          name: 'Positron'
+          className: 'positron_rainbow'
+          attribution: '© <a href="http://www.openstreetmap.org/copyright">OpenStreetMap</a> contributors © <a href= "https://carto.com/attributions">CARTO</a>'
+        dark_matter_rainbow:
+          url: 'http://{s}.basemaps.cartocdn.com/dark_all/{z}/{x}/{y}.png'
+          subdomains: 'abcd'
+          minZoom: '0'
+          maxZoom: '18'
+          name: 'Dark matter'
+          className: 'dark_matter_rainbow'
+          attribution: '© <a href="http://www.openstreetmap.org/copyright">OpenStreetMap</a> contributors © <a href="https://carto.com/attributions">CARTO</a>'
+        positron_lite_rainbow:
+          url: 'http://{s}.basemaps.cartocdn.com/light_nolabels/{z}/{x}/{y}.png'
+          subdomains: 'abcd'
+          minZoom: '0'
+          maxZoom: '18'
+          name: 'Positron (lite)'
+          className: 'positron_lite_rainbow'
+          attribution: '© <a href="http://www.openstreetmap.org/copyright">OpenStreetMap</a> contributors © <a href="https://carto.com/attributions">CARTO</a>'
+
+      stamen:
+        toner_stamen:
+          url: 'https://stamen-tiles-{s}.a.ssl.fastly.net/toner/{z}/{x}/{y}.png'
+          subdomains: 'abcd'
+          minZoom: '0'
+          maxZoom: '18'
+          name: 'Toner'
+          className: 'toner_stamen'
+          attribution: 'Map tiles by <a href="http://stamen.com">Stamen Design</a>, under <a href="http://creativecommons.org/licenses/by/3.0">CC BY 3.0</a>. Data by <a href="http://openstreetmap.org">OpenStreetMap</a>, under <a href="http://www.openstreetmap.org/copyright">ODbL</a>.'
+
+Basemaps with a layer of labels
+-------------------------------
+Basemaps can optionally add a layer with labels on top of other layers. To do so,
+you should add the labels key to the basemap config, as follows:
+
+.. code-block:: yaml
+
+  positron_rainbow:
+    default: true
+    url: 'http://{s}.basemaps.cartocdn.com/light_all/{z}/{x}/{y}.png'
+    subdomains: 'abcd'
+    minZoom: '0'
+    maxZoom: '18'
+    name: 'Positron'
+    className: 'positron_rainbow'
+    attribution: '© <a href="http://www.openstreetmap.org/copyright">OpenStreetMap</a> contributors © <a href= "https://carto.com/attributions">CARTO</a>'
+    labels:
+      url: 'http://{s}.basemaps.cartocdn.com/light_only_labels/{z}/{x}/{y}.png'
+
+Domainless URLs
+---------------
+
+Historically, CARTO URLs were based on a ``username.carto.com/PATH`` schema.
+When Multiuser accounts were introduced, an alternate schema
+``organizationname.carto.com/u/username/PATH`` was built alongside the "classic" one.
+Both schemas introduce some problems for opensource and/or custom installs of the platform,
+as they require DNS changes each time a new user or organization is added.
+
+Subdomainless urls are the answer to this problems. Modifying some configuration settings,
+any CARTO installation can be setup to work with a new schema, ``carto.com/user/username/PATH.``
+
+The following sections details the steps to make it work and the limitations it has.
+
+Configuration changes for Domainless URLs
+------------------------------------------
+
+* For a default installation, **app_config.yml** contains this relevant values:
+
+  .. code-block:: yaml
+
+    session_domain:     '.localhost.lan'
+    subdomainless_urls: false
+
+* To activate subdomainless urls, change to (notice the removed starting dot from session_domain:
+
+  .. code-block:: yaml
+
+    session_domain:     'localhost.lan'
+    subdomainless_urls: true
+
+
+* Non-default HTTP and HTTPs ports can also be configured here for REST API calls, with the following **app_config.yml** attributes:
+
+  .. code-block:: yaml
+
+    # nil|integer. HTTP port to use when building urls.
+    # Leave empty to use default (80)
+    http_port:
+    # nil|integer. HTTPS port to use when building urls.
+    # Leave empty to use default (443)
+    https_port:
+
+Remember that as with other configuration changes, Rails application must be restarted to apply them.
+
+Limitations
+-----------
+If you leave the dot at ``session_domain`` having subdomainless urls, you will be forced
+to always have a subdomain. Any will do, but must be present. If you remove the dot it
+will work as intended without any subdomain.
+
+When subdomainless urls are used, organizations will be ignored from the urls. In fact,
+typing ``whatever.carto.com/user/user1`` and ``carto.com/user/user1`` is the same. The platform
+will replicate the sent subdomain fragment to avoid CORS errors but no existing organization
+checks will be performed. You should be able to use them, assign quota to the organization users, etc.
+
+Common Data
+-----------
+This service uses the visualizations API to retrieve all the public datasets from a defined user and
+serve them as importable datasets to all the users of the platform through the data library options.
+
+All can be configured through the ``common_data`` settings section. If the ``base_url``
+option is set, this will be the base url the service is going to use to build the URL to retrieve datasets.
+For example:
+
+.. code-block:: yaml
+
+  common_data:
+    protocol: 'https'
+    username: 'common-data'
+    base_url: 'https://common-data.carto.com'
+    format: 'shp'
+
+
+Use ``https://common-data.carto.com`` as the base url to retrieve all the public datasets from that user.
+
+This is the default behaviour in CARTO, but if you want to use your own system and user for this purpose you
+have to define the ``username`` property pointing to the user that will provide the datasets in your own instance.
+The URL in this case is going to be built using your instance base url. For example if your instance base url is
+``http://www.example.com`` and the config is:
+
+.. code-block:: yaml
+
+  common_data:
+    protocol: 'https'
+    username: 'common-data-user'
+    format: 'shp'
+
+the system populates the data library with the public datasets from ``http://common-data-user.example.com...``
+
+The ``format`` option is used to define the format of the file generated when you are importing one datasets from
+the data library. When you import a dataset it uses a stored URL to download that dataset as a file, in the format
+defined in the config, and import as your own dataset.
+
+Separate folders
+----------------
+
+Default installation keeps logs, configuration files and assets under the standard Rails folder structure: ``/log``,
+``/config`` and ``/public`` at Rails root (your installation directory). Some installations might be interested in
+moving those directories outside Rails root in order to separate code and data. You can accomplish that with symbolic
+links. Nevertheless, there are three environment variables that you can use instead:
+
+* ``RAILS_LOG_BASE_PATH``: for example, setting it to ``/var/carto`` will use that as a base folder for log files, which
+  will be stored at ``/var/carto/log``. Defaults to ``Rails.root``.
+* ``RAILS_CONFIG_BASE_PATH``: for example, setting it to ``/etc/carto`` will make Rails open the application and database
+  configuration files at ``/etc/carto/conf/app_config.yml`` and ``/etc/carto/conf/database.yml``. Defaults to ``Rails.root``.
+* ``RAILS_PUBLIC_UPLOADS_PATH``: sets assets base path, both static and dynamic. For example, setting
+  it to ``/var/carto/assets`` will upload files (markers, avatars and so on) to ``/var/carto/assets/uploads``, but it also
+  makes Rails server to load public assets (CSSs, JS...) from there. Defaults to ``app_config[:importer]["uploads_path"]`` or ``Rails.root``
+  if it's not present (due to backwards compatibility).  If you use this variable you'll need to do one onf the following:
+
+  * Use nginx to load the assets (recommended): making ``/public`` the nginx default root will make nginx use the proper
+    folders for assets, without requesting them to the Rails server: ``root /opt/carto/builder/embedded/cartodb/public;``.
+  * Copy or link assets (from ``/<RAILS ROOT>/public``) to public upload path folder.

doc/manual/source/index.rst

@@ -0,0 +1,29 @@
+.. CARTO Platform Documentation documentation master file, created by
+   sphinx-quickstart on Wed Jun 24 10:50:07 2015.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Welcome to CARTO Manual
+=======================
+
+Contents:
+
+.. toctree::
+   :maxdepth: 2
+
+   intro
+   components
+   install
+   run
+   configuration
+   operations
+
+
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`
+

doc/manual/source/install.rst

@@ -0,0 +1,393 @@
+
+Installation
+============
+
+.. warning::
+  CARTO works with Ubuntu 16.04 x64. This documentation describes the process to install CartoDB in this specific OS version.
+
+  However this doesn't mean that it won't work with other Operating Systems or other Ubuntu. There are also many successful installations on Amazon EC2, Linode, dedicated instances and development machines running OS X and Ubuntu 12.04+.
+
+  You will find notes along this guide explaining some of the Ubuntu 16.04 specifics, and pointing to alternative solutions for other environments.
+
+System requirements
+-------------------
+Besides the OS version mentioned in the introduction, there are some system requirements needed before starting with the installation of the stack. Also this process assumes that you have enough permissions in the system to run successfully most part of the commands of this doc.
+
+System locales
+~~~~~~~~~~~~~~
+
+Installations assume you use UTF8. You can set the locale by doing this:
+
+.. code-block:: bash
+
+  sudo locale-gen en_US.UTF-8
+  sudo update-locale LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
+
+
+Build essentials
+~~~~~~~~~~~~~~~~
+
+Although we try to maintain packaged versions of almost every part of the stack, there are some parts like gems or npm packages that need some development tools in the system in order to compile. You can install all the needed build tools by doing this:
+
+.. code-block:: bash
+
+  sudo apt-get install make pkg-config
+
+
+GIT
+~~~
+
+You will need git commands in order to handle some repositories and install some dependencies:
+
+.. code-block:: bash
+
+  sudo apt-get install git
+
+PostgreSQL
+----------
+
+.. note::
+  CARTO requires PostgreSQL 10+. The PPA packages also provide some additional patches, which are not needed but help improve the experience in production environments.
+
+* Add PPA repository
+
+  .. code-block:: bash
+
+    sudo add-apt-repository ppa:cartodb/postgresql-10 && sudo apt-get update
+
+* Install packages
+
+  .. code-block:: bash
+
+    sudo apt-get install postgresql-10 \
+                         postgresql-plpython-10 \
+                         postgresql-server-dev-10
+
+
+PostgreSQL access authorization is managed through pg_hba.conf configuration file, which is normally in ``/etc/postgresql/10/main/pg_hba.conf``. Here it's defined how the users created in postgresql cluster can access the server. This involves several aspects like type of authentication (md5, no password, etc..) or source IP of the connection. In order to simplify the process of the installation we are going to allow connections with postgres user from localhost without authentication. Of course this can be configured in a different way at any moment but changes here should imply changes in database access configuration of CARTO apps.
+
+Edit ``/etc/postgresql/10/main/pg_hba.conf``, modifying the existing lines to use ``trust`` authentication (no password access from localhost):
+
+  .. code-block:: bash
+
+    local   all             postgres                                trust
+    local   all             all                                     trust
+    host    all             all             127.0.0.1/32            trust
+
+For these changes to take effect, you'll need to restart postgres:
+
+  .. code-block:: bash
+
+    sudo systemctl restart postgresql
+
+
+* Create some users in PostgreSQL. These users are used by some CARTO apps internally
+
+  .. code-block:: bash
+
+    sudo createuser publicuser --no-createrole --no-createdb --no-superuser -U postgres
+    sudo createuser tileuser --no-createrole --no-createdb --no-superuser -U postgres
+
+* Install CartoDB postgresql extension. This extension contains functions that are used by different parts of the CartoDB platform, included Builder and the SQL and Maps API.
+
+  .. code-block:: bash
+
+    git clone https://github.com/CartoDB/cartodb-postgresql.git
+    cd cartodb-postgresql
+    git checkout <LATEST cartodb-postgresql tag>
+    sudo make all install
+
+GIS dependencies
+----------------
+
+* Add GIS PPA
+
+  .. code-block:: bash
+
+    sudo add-apt-repository ppa:cartodb/gis && sudo apt-get update
+
+
+* Install GDAL
+
+  .. code-block:: bash
+
+    sudo apt-get install gdal-bin libgdal-dev
+
+PostGIS
+-------
+
+.. note::
+  CARTO requires PostGIS 2.4. The PPA just packages this version for Ubuntu 16.04.
+
+* Install PostGIS
+
+  .. code-block:: bash
+
+    sudo apt-get install postgis
+
+* Initialize template postgis database. We create a template database in postgresql that will contain the postgis extension. This way, every time CartoDB creates a new user database it just clones this template database
+
+  .. code-block:: bash
+
+    sudo createdb -T template0 -O postgres -U postgres -E UTF8 template_postgis
+    psql -U postgres template_postgis -c 'CREATE EXTENSION postgis;CREATE EXTENSION postgis_topology;'
+    sudo ldconfig
+
+* (Optional) Run an installcheck to verify the database has been installed properly
+
+  .. code-block:: bash
+
+   sudo PGUSER=postgres make installcheck # to run tests
+
+  Check https://github.com/cartodb/cartodb-postgresql for further reference
+
+
+Redis
+-----
+
+.. note::
+    CARTO requires Redis 4+. You can also optionally install redis-cell for rate limiting, which is not described by this guide.
+
+* Add redis PPA
+
+  .. code-block:: bash
+
+    sudo add-apt-repository ppa:cartodb/redis-next && sudo apt-get update
+
+* Install redis
+
+  .. code-block:: bash
+
+    sudo apt-get install redis
+
+.. warning::
+
+  By default redis server is configured to only have periodic snapshotting to disk. If stopped or restarted some data stored in redis since the last snahpshot can be lost. In CARTO redis is not just a simple cache storage. It stores information that need to be persisted.
+
+  For data safety, make sure to have proper values of *save*, *appendonly* and *appendfsync* config attributes. For more information check `http://redis.io/topics/persistence`
+
+Node.js
+-------
+
+.. note::
+    CARTO requires Node.js 10+ and npm 6+.
+
+Node.js is required by different parts of the stack. The more significant are the Maps and SQL APIs. It's also used to install and execute some dependencies of Builder.
+
+* Install Node.js
+
+  .. code-block:: bash
+
+    curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
+    sudo apt-get install -y nodejs
+
+  Note this should install both Node.js 10.x and npm 6.x. You can verify the installation went as expected with:
+
+  .. code-block:: bash
+
+    node -v
+    npm -v
+
+We will also install some development libraries that will be necessary to build some Node.js modules:
+
+  .. code-block:: bash
+
+    sudo apt-get install libpixman-1-0 libpixman-1-dev
+    sudo apt-get install libcairo2-dev libjpeg-dev libgif-dev libpango1.0-dev
+
+SQL API
+-------
+
+* Download API
+
+  .. code-block:: bash
+
+    git clone git://github.com/CartoDB/CartoDB-SQL-API.git
+    cd CartoDB-SQL-API
+
+* Install npm dependencies
+
+  .. code-block:: bash
+
+    npm install
+
+* Create configuration. The name of the filename of the configuration must be the same than the environment you are going to use to start the service. Let's assume it's development.
+
+  .. code-block:: bash
+
+    cp config/environments/development.js.example config/environments/development.js
+
+
+* Start the service. The second parameter is always the environment if the service. Remember to use the same you used in the configuration.
+
+  .. code-block:: bash
+
+    node app.js development
+
+
+MAPS API
+--------
+
+* Download API
+
+  .. code-block:: bash
+
+    git clone git://github.com/CartoDB/Windshaft-cartodb.git
+    cd Windshaft-cartodb
+
+* Install yarn dependencies
+
+  .. code-block:: bash
+
+    npm install
+
+* Create configuration. The name of the filename of the configuration must be the same than the environment you are going to use to start the service. Let's assume it's development.
+
+  .. code-block:: bash
+
+    cp config/environments/development.js.example config/environments/development.js
+    mkdir logs
+
+
+* Start the service. The second parameter is always the environment of the service. Remember to use the same you used in the configuration.
+
+  .. code-block:: bash
+
+    node app.js development
+
+
+Ruby
+----
+
+.. note::
+  CARTO requires exactly Ruby 2.4.x. Older or newer versions won't work.
+
+* Add brightbox ruby repositories
+
+  .. code-block:: bash
+
+    sudo apt-add-repository ppa:brightbox/ruby-ng && sudo apt-get update
+
+* Install ruby 2.4
+
+  .. code-block:: bash
+
+    sudo apt-get install ruby2.4 ruby2.4-dev
+
+* Install bundler. Bundler is an app used to manage ruby dependencies. It is needed by CARTO Builder
+
+  .. code-block:: bash
+
+    sudo apt-get install ruby-bundler
+
+
+* Install compass. It will be needed later on by CARTO's Builder
+
+  .. code-block:: bash
+
+    sudo gem install compass
+
+
+Builder
+-------
+
+.. note::
+  CARTO users Python 2.7+. Python 3 will not work correctly.
+
+* Download Builder's code
+
+  .. code-block:: bash
+
+    git clone --recursive https://github.com/CartoDB/cartodb.git
+    cd cartodb
+
+* Install pip
+
+  .. code-block:: bash
+
+   sudo apt-get install python-pip
+
+
+* Install ruby dependencies
+
+  .. code-block:: bash
+
+    sudo apt-get install imagemagick unp zip libicu-dev
+    RAILS_ENV=development bundle install
+
+
+* Install python dependencies
+
+  .. code-block:: bash
+
+    sudo pip install --no-use-wheel -r python_requirements.txt
+
+.. warning::
+    If this fails due to the installation of the gdal package not finding Python.h or any other header file, you'll need to do this:
+
+    ::
+
+        export CPLUS_INCLUDE_PATH=/usr/include/gdal
+        export C_INCLUDE_PATH=/usr/include/gdal
+        export PATH=$PATH:/usr/include/gdal
+
+    After this, re-run the pip install command. Variables can be passed to sudo if exporting them and re-running ``pip install`` doesn't work:
+
+    .. code-block:: bash
+
+       sudo CPLUS_INCLUDE_PATH=/usr/include/gdal C_INCLUDE_PATH=/usr/include/gdal PATH=$PATH:/usr/include/gdal pip install --no-use-wheel -r python_requirements.txt
+
+    If gdal keeps failing, see more information here: http://gis.stackexchange.com/questions/28966/python-gdal-package-missing-header-file-when-installing-via-pip
+
+* Install Node.js dependencies
+
+  .. code-block:: bash
+
+    npm install
+
+
+* Compile static assets
+
+  .. code-block:: bash
+
+    npm run carto-node && npm run build:static
+
+* (Optional) Precompile assets. Needed if you don't want to use CARTO's CDN for assets.
+
+  .. code-block:: bash
+
+    export PATH=$PATH:$PWD/node_modules/grunt-cli/bin
+    bundle exec grunt --environment=development
+
+
+* Create configuration files
+
+  .. code-block:: bash
+
+    cp config/app_config.yml.sample config/app_config.yml
+    cp config/database.yml.sample config/database.yml
+
+* Start the redis-server that allows access to the SQL and Maps APIs:
+
+  .. code-block:: bash
+
+    sudo systemctl start redis-server
+
+* Initialize the metadata database
+
+  .. code-block:: bash
+
+    RAILS_ENV=development bundle exec rake db:create
+    RAILS_ENV=development bundle exec rake db:migrate
+
+* Start Builder's HTTP server
+
+  .. code-block:: bash
+
+    RAILS_ENV=development bundle exec rails server
+
+* In a different process/console start the resque process
+
+  .. code-block:: bash
+
+    RAILS_ENV=development bundle exec ./script/resque

doc/manual/source/intro.rst

@@ -0,0 +1,44 @@
+CARTO introduction
+==================
+
+CARTO is an open source tool that allows for the storage and
+visualization of geospatial data on the web.
+
+It was built to make it easier for people to tell their stories by
+providing them with flexible and intuitive ways to create maps and design
+geospatial applications. CARTO can be installed on your own server
+and we also offer a hosted service at `carto.com <https://carto.com>`_.
+
+If you would like to see some live demos, check out our
+`videos <http://vimeo.com/channels/cartodb>`_ on Vimeo.
+We hope you like it!
+
+.. image:: static/map_view.png
+   :alt: Map View
+   :align: center
+
+What can I do with CARTO?
+----------------------------
+
+With CARTO, you can upload your geospatial data (Shapefiles, GeoJSON,
+etc) using a web form and then make it public or private.
+
+After it is uploaded, you can visualize it in a table or on a map, search
+it using SQL, and apply map styles using CartoCSS. You can even access it
+using the CARTO `API OVERVIEW <https://docs.carto.com/cartodb-platform.html>`_
+and `SQL API <https://docs.carto.com/cartodb-platform/sql-api.html>`_, or export it
+to a file.
+
+In other words, with CARTO you can make awesome maps and build
+powerful geospatial applications! Definitely check out the `CARTO
+Help Center <https://carto.com/help/>`_ for interactive examples
+and code.
+
+.. image:: static/map_view_wizard.png
+   :alt: Map View Wizard
+   :align: center
+
+
+.. image:: static/data_view.png
+   :alt: Data View
+   :align: center

doc/manual/source/operations.rst

@@ -0,0 +1,17 @@
+Operations
+==========
+
+Certain operations in the CARTO platform can be done using rake tasks or scripts that bundle several of them together. In this section you will instructions to carry out various common operations.
+
+Common operations in CARTO include:
+
+.. toctree::
+
+  operations/create_users.rst
+  operations/create_orgs.rst
+  operations/change_feature_flags.rst
+  operations/changing_limits.rst
+  operations/exporting_importing_visualizations.rst
+  operations/run_sync_tables.rst
+  operations/http_headers_authentication
+  operations/configure_data_services.rst

doc/manual/source/operations/change_feature_flags.rst

@@ -0,0 +1,200 @@
+Changing Feature Flags
+======================
+
+CARTO uses feature flags, so different users can have access to different features of CARTO. If you would like to enable or disable feature flags to one or all users or to a given organization, you can use the rake tasks described in this section. Feature flag creation and deletion are also covered.
+
+
+Enabling a feature for all users
+--------------------------------
+
+Enabling a feature for all users is done with a rake task called ``enable_feature_for_all_users`` and it takes one parameter.
+
+* ``feature_flag_name`` is the name of the feature flag to be enabled. For example: 'special_dashboard'.
+
+This task is executed like:
+
+.. code-block:: bash
+
+  $ bundle exec rake cartodb:features:enable_feature_for_all_users[<feature_flag_name>]
+
+And an example to enable the 'special_dashboard' feature could be:
+
+.. code-block:: bash
+
+  $ bundle exec rake cartodb:features:enable_feature_for_all_users["special_dashboard"]
+
+
+Enabling a feature for a given user
+-----------------------------------
+
+Enabling a feature for a given user is done with a rake task called ``enable_feature_for_user`` and it takes two parameters.
+
+* ``feature_flag_name`` is the name of the feature flag to be enabled. For example: 'special_dashboard'.
+* ``user_name`` is the user name of the user to whom the feature flag is to be enabled. For example: 'manolo'.
+
+This task is executed like:
+
+.. code-block:: bash
+
+  $ bundle exec rake cartodb:features:enable_feature_for_user[<feature_flag_name>,<user_name>]
+
+.. warning::
+
+  Please be very careful **NOT** to leave a space between parameters, as it will cause rake to spit a ``don't know how to build task`` type error.
+
+And an example to enable the 'special_dashboard' feature for user with user name 'manolo' could be:
+
+.. code-block:: bash
+
+  $ bundle exec rake cartodb:features:enable_feature_for_user["special_dashboard","manolo"]
+
+
+Enabling a feature for a given organization
+-------------------------------------------
+
+Enabling a feature for a given organization is done with a rake task called ``enable_feature_for_organization`` and it takes two parameters.
+
+* ``feature_flag_name`` is the name of the feature flag to be enabled. For example: 'special_dashboard'.
+* ``organization_name`` is the internal name ('cartodb' vs 'CartoDB Inc.') to which the feature flag is to be enabled. For example: 'cartodb'.
+
+This task is executed like:
+
+.. code-block:: bash
+
+  $ bundle exec rake cartodb:features:enable_feature_for_organization[<feature_flag_name>,<organization_name``
+
+.. warning::
+
+  Please be very careful **NOT** to leave a space between parameters, as it will cause rake to spit a ``don't know how to build task`` type error.
+
+And an example to enable the 'special_dashboard' feature for organization 'cartodb' could be:
+
+.. code-block:: bash
+
+  $ bundle exec rake cartodb:features:enable_feature_for_organization["special_dashboard","cartodb"]
+
+
+Disabling a feature for all users
+---------------------------------
+
+Disabling a feature for all users is done with a rake task called ``disable_feature_for_all_users`` and it takes one parameter.
+
+* ``feature_flag_name`` is the name of the feature flag to be disabled. For example: 'special_dashboard'.
+
+This task is executed like:
+
+.. code-block:: bash
+
+  $ bundle exec rake cartodb:features:disable_feature_for_all_users[<feature_flag_name>]
+
+And an example to disable the 'special_dashboard' feature could be:
+
+.. code-block:: bash
+
+  $ bundle exec rake cartodb:features:disable_feature_for_all_users["special_dashboard"]
+
+
+Disabling a feature for a given user
+------------------------------------
+
+Disabling a feature for a given user is done with a rake task called ``disable_feature_for_user`` and it takes two parameters.
+
+* ``feature_flag_name`` is the name of the feature flag to be disabled. For example: 'special_dashboard'.
+* ``user_name`` is the user name of the user to whom the feature flag is to be disabled. For example: 'manolo'.
+
+This task is executed like:
+
+.. code-block:: bash
+
+  $ bundle exec rake cartodb:features:disable_feature_for_user[<feature_flag_name>,<user_name>]
+
+.. warning::
+
+  Please be very careful **NOT** to leave a space between parameters, as it will cause rake to spit a ``don't know how to build task`` type error.
+
+And an example to disable the 'special_dashboard' feature for user with user name 'manolo' could be:
+
+.. code-block:: bash
+
+  $ bundle exec rake cartodb:features:disable_feature_for_user["special_dashboard","manolo"]
+
+
+Disabling a feature for a given organization
+--------------------------------------------
+
+Disabling a feature for a given organization is done with a rake task called ``disable_feature_for_organization`` and it takes two parameters.
+
+* ``feature_flag_name`` is the name of the feature flag to be disabled. For example: 'special_dashboard'.
+* ``organization_name`` is the internal name ('cartodb' vs 'CartoDB Inc.') to which the feature flag is to be disabled. For example: 'cartodb'.
+
+This task is executed like:
+
+.. code-block:: bash
+
+  $ bundle exec rake cartodb:features:disable_feature_for_organization[<feature_flag_name>,<organization_name``
+
+.. warning::
+
+  Please be very careful **NOT** to leave a space between parameters, as it will cause rake to spit a ``don't know how to build task`` type error.
+
+And an example to disable the 'special_dashboard' feature for organization 'cartodb' could be:
+
+.. code-block:: bash
+
+  $ bundle exec rake cartodb:features:disable_feature_for_organization["special_dashboard","cartodb"]
+
+
+Adding a feature flag
+---------------------
+
+Adding feature flags should be done using the rake task called ``add_feature_flag``. This rake task only takes one argument:
+
+* ``feature_flag_name`` is the name of the feature flag to be created.
+
+This task is executed like:
+
+.. code-block:: bash
+
+  $ bundle exec rake cartodb:features:add_feature_flag[<feature_flag_name>]
+
+And an example to create a feature flag named "special_dashboard" could be:
+
+.. code-block:: bash
+
+  $ bundle exec rake cartodb:features:add_feature_flag["special_dashboard"]
+
+
+Removing a feature flag
+-----------------------
+
+Removing feature flags should be done using the rake task called ``remove_feature_flag``. This rake task only takes one argument:
+
+* ``feature_flag_name`` is the name of the feature flag to be removed.
+
+This task is executed like:
+
+.. code-block:: bash
+
+  $ bundle exec rake cartodb:features:remove_feature_flag[<feature_flag_name>]
+
+And an example to remove a feature flag named "special_dashboard" could be:
+
+.. code-block:: bash
+
+  $ bundle exec rake cartodb:features:remove_feature_flag["special_dashboard"]
+
+
+Listing all feature flags
+-------------------------
+
+All existing feature flags can be listed using the rake task called ``list_all_features``.
+
+This task is executed like:
+
+.. code-block:: bash
+
+  $ bundle exec rake cartodb:features:list_all_features
+
+
+
+

doc/manual/source/operations/changing_limits.rst

@@ -0,0 +1,34 @@
+Changing limits
+===============
+
+This section explains how to use rake tasks to change several limits for users.
+
+Change user import limits
+-------------------------
+
+Using the rake task ``set_custom_limits_for_user``, you can change import limits for a given user. The parameters this task takes are:
+
+* ``user_name`` is the user name of the user for whom these limits will be changed.
+* ``import_file_size`` is the maximum size in **bytes** for a file to be imported.
+* ``table_row_count`` is the maximum number of rows for a file to be imported.
+* ``concurrent_imports`` is the maximum number of concurrent imports that can be handled at once.
+
+
+
+This task is executed like:
+
+.. code-block:: bash
+
+  $ bundle exec rake cartodb:set_custom_limits_for_user[<user_name>,<import_file_size>,<table_row_count>,<concurrent_imports>]
+
+and an example execution could be:
+
+.. code-block:: bash
+
+  $ bundle exec rake cartodb:set_custom_limits_for_user["manolo","1048576","50000","5"]
+
+Increasing Twitter imports limit
+--------------------------------
+
+Increasing the Twitter imports limit should done using rake task ``increase_limits_for_twitter_import_users``. This rake task takes no parameters. Upon execution, all users with Twitter imports enabled will have **1500MB of file size quota** and a **5M row quota** limit.
+

doc/manual/source/operations/configure_data_services.rst

@@ -0,0 +1,125 @@
+Configuring Dataservices
+========================
+
+The services provided by the `Dataservices SQL extension <https://github.com/CartoDB/dataservices-api>`_
+can be manually configured for users or organizations through the following rake tasks.
+
+The service configuration is stored in the users and organization metadata tables and reflected in the REDIS configuration database.
+
+Service provider
+----------------
+
+For each basic service class (``geocoder``, ``routing``, ``isolines``) a provider can be assigned with the tasks:
+
+* ``cartodb:services:set_user_provider[username,service,provider]``
+* ``cartodb:services:set_organization_provider[orgname,service,provider]``
+
+Valid providers are:
+
+* ``mapzen`` for Mapzen-based third party services (Mapzen Search/Mapzen Mobility)
+* ``here`` for HERE maps third party services (HERE Geocoder/Routing APIs)
+* ``google`` for Google Maps services
+
+Examples:
+`````````
+
+Set geocoder provider for user ``user-name`` to Mapzen::
+
+    rake cartodb:services:set_user_provider['user-name',geocoder,mapzen]
+
+Set geocoder provider for organization ``org-name`` to Mapzen::
+
+    rake cartodb:services:set_organization_provider['org-name',geocoder,mapzen]
+
+Quotas
+------
+
+Service limits can be established for individual users or organizations in the form of quotas (maximum number of requests per billing period).
+The next tasks manage the configuration of the quotas.
+
+* ``cartodb:services:set_user_quota[username,service,quota]``
+* ``cartodb:services:set_org_quota[orgname,service,quota]``
+
+Quota values must be non-negative integers.
+
+Valid services are:
+
+* ``geocoding`` for street-level (hi-res) geocoding services.
+* ``here_isolines`` for general isolines/isochrones zoning services (despite the name, this does not only applies to Here-provided services).
+* ``obs_snapshot`` for Data Observatory *snapshot* services.
+* ``obs_general`` for Data Observatory general services.
+* ``mapzen_routing`` for general routing services (again, not necessarily provided by Mapzen).
+
+
+Examples:
+`````````
+
+Set geocoding quota for user ``user-name`` to 1000 monthly requests::
+
+    rake cartodb:services:set_user_quota['user-name',geocoder,1000]
+
+Set geocoder quota for organization ``org-name`` to 1000 monthly requests::
+
+    rake cartodb:services:set_org_quota['org-name',geocoder,1000]
+
+Soft limits
+-----------
+
+The service limits for a user can be configured to be *soft*, meaning that the user can exceed the limits (possibly incurring in additional charges).
+So, when soft limits is set to `false` (the default) and the limits are exceeded, service requests will fail,
+while if the soft limits are set to `true` requests will succeed (and the user will be charged for the excess).
+
+* ``cartodb:services:set_user_soft_limit[username,service,soft_limit_status]``
+
+The possible values for the soft limits status is either ``true`` or ``false``.
+
+Valid services are, as for the quota configuration:
+
+* ``geocoding`` for street-level (hi-res) geocoding services.
+* ``here_isolines`` for general isolines/isochrones zoning services (despite the name, this does not only applies to Here-provided services).
+* ``obs_snapshot`` for Data Observatory *snapshot* services.
+* ``obs_general`` for Data Observatory general services.
+* ``mapzen_routing`` for general routing services (again, not necessarily provided by Mapzen).
+
+Examples:
+`````````
+
+Activate soft geocoding limits for user ``user-name``::
+
+    rake cartodb:services:set_user_soft_limit['user-name',geocoder,true]
+
+Disable soft geocoding limits for user ``user-name``::
+
+    rake cartodb:services:set_user_soft_limit['user-name',geocoder,false]
+
+User database configuration (from Builder)
+------------------------------------------
+
+Add the following entry to the `geocoder` entry of the `cartodb/config/app_config.yml` file:
+
+.. code-block:: yaml
+
+  api:
+      host: 'localhost'
+      port: '5432'
+      user: 'dataservices_user'
+      dbname: 'dataservices_db'
+
+
+In the `cartodb/config/app_config.yml` file, enable the desired dataservices:
+
+.. code-block:: yaml
+
+   enabled:
+       geocoder_internal: false
+       hires_geocoder: false
+       isolines: false
+       routing: false
+       data_observatory: true
+
+
+Execute the rake tasks to update all the users and organizations:
+
+``bundle exec rake cartodb:db:configure_geocoder_extension_for_organizations['', true]``
+
+``bundle exec rake cartodb:db:configure_geocoder_extension_for_non_org_users['', true]``

doc/manual/source/operations/create_orgs.rst

@@ -0,0 +1,34 @@
+Creating organizations
+======================
+
+To create a new organization, a rake task called ``create_new_organization_with_owner`` is used. For this task to work properly, a user **created beforehand** must be provided as an owner (if you're not sure how to create a user, refer to "Creating Users").
+
+In order to create the organization, 4 parameters must be set:
+
+* ``ORGANIZATION_NAME`` is a short nickname for the organization, that may only contain letters, numbers and dash (-) characters. For example, 'cartodb' would be OK.
+* ``ORGANIZATION_DISPLAY_NAME`` is a longer, more beautiful name for the organization. It may contain any characters needed. For example, 'CartoDB Inc.'.
+* ``ORGANIZATION_SEATS`` is the number of users that will be able to be created under the organization. For example, 5 seats will mean that a maximum of 5 users can belong to the organization.
+* ``ORGANIZATION_QUOTA`` is the space quota in **bytes** that the organization is assigned. For example, 1024 * 1024 * 1024 is 1GB of quota.
+* ``USERNAME`` is the user name of the owner of the organization. In our example, let's assume that our user name is 'manolo'.
+
+This task is executed like:
+
+.. code-block:: bash
+
+  $ bundle exec rake cartodb:db:create_new_organization_with_owner ORGANIZATION_NAME="<org_name>" ORGANIZATION_DISPLAY_NAME="<org_display_name>" ORGANIZATION_SEATS="<org_seats>" ORGANIZATION_QUOTA="<org_quota>" USERNAME="<username>"
+
+
+and an example execution for creating an organization owned by 'manolo', named 'CartoDB Inc.', referred to as 'cartodb', with 5 seats and a 1GB quota, would be:
+
+.. code-block:: bash
+
+  $ bundle exec rake cartodb:db:create_new_organization_with_owner ORGANIZATION_NAME="cartodb" ORGANIZATION_DISPLAY_NAME="CartoDB Inc." ORGANIZATION_SEATS="5" ORGANIZATION_QUOTA="1073741824" USERNAME="manolo"
+
+Seats
+-----
+
+You can change the viewer seats:
+
+.. code-block:: bash
+
+  $ bundle exec rake cartodb:db:set_organization_viewer_seats["<org_name>","<viewer_seats>"]

doc/manual/source/operations/create_users.rst

@@ -0,0 +1,15 @@
+Creating users
+==============
+
+Creating users in CARTO is simple thanks to the ``create_dev_user`` script located in ``scripts/create_dev_user``. To execute this script, be sure to be located at the cartodb repository root directory and simply run:
+
+.. code-block:: bash
+
+  $ ./script/create_dev_user
+
+You will be prompted to input two parameters:
+
+* **subdomain** is the same as the user's user name. This is what you will enter in the browser to access the user's dashboard: ``https://<user_name>.carto.com``. Set it to whatever you want the user's user name to be.
+* **password** this is the password the new user will use to login into their account.
+
+Upon script completion, the new user will have been created.

doc/manual/source/operations/exporting_importing_visualizations.rst

@@ -0,0 +1,37 @@
+Exporting/Importing visualizations
+==================================
+
+You might be interested in exporting and importing visualizations because of several reasons:
+
+* Backup purposes.
+* Moving visualizations between different hosts.
+* Moving visualizations between users.
+* Using the same visualization with different data.
+
+With ``cartodb:vizs:export_user_visualization_json`` task you can export a visualization to JSON, and with ``cartodb:vizs:import_user_visualization_json`` you can import it. First outputs to stdout and second reads stdin.
+
+This example exports ``c54710aa-ad8f-11e5-8046-080027880ca6`` visualization.
+
+.. code-block:: bash
+
+  $ bundle exec rake cartodb:vizs:export_user_visualization_json['c54710aa-ad8f-11e5-8046-080027880ca6'] > c54710aa-ad8f-11e5-8046-080027880ca6.json
+
+and this imports it into ``6950b745-5524-4d8d-9478-98a8a04d84ba`` user, who is in another server.
+
+.. code-block:: bash
+
+  $ cat c54710aa-ad8f-11e5-8046-080027880ca6.json | bundle exec rake cartodb:vizs:import_user_visualization_json['6950b745-5524-4d8d-9478-98a8a04d84ba']
+
+Please keep in mind the following:
+
+* Exporting has **backup purposes**, so it keeps ids. If you want to use this to replicate a visualization in the same server you can edit the JSON and change the ids. Any valid, distinct UUID will work.
+* It **does export neither the tables nor its data**. Destination user should have tables with the same name than the original one for the visualization to work. You can change the table names in the JSON file if names are different.
+
+Exporting/Importing full visualizations
+=======================================
+
+*Disclaimer: this feature is still in beta*
+
+You can export a complete visualization (data, metadata and map) with this command: ``bundle exec rake cartodb:vizs:export_full_visualization['5478433b-b791-419c-91d9-d934c56f2053']``
+
+That will generate a `.carto` file that you can import in any CARTO installation just dropping the file as usual.

doc/manual/source/operations/http_headers_authentication.rst

@@ -0,0 +1,34 @@
+HTTP Header Authentication
+==================================
+
+With web servers such as NGINX or others you can perform SSO by making the web server add a trusted, safe header to every request sent to CARTO. Example:
+
+User browser -- ``GET http://myorg.mycompany.lan/dashboard`` --> NGINX (adds ``'sso-user-email': 'alice@myorg.com'`` header) --> CARTO server
+
+You can enable HTTP Header Authentication at CARTO by adding the following to ``app_conf.yml`` (taken from ``app_conf.yml.sample``):
+
+.. code-block:: Ruby
+
+  http_header_authentication:
+    header: # name of the trusted, safe header that your server adds to the request
+    field: # 'email' / 'username' / 'id' / 'auto' (autodetection)
+    autocreation: # true / false (true requires field to be email)
+
+Configuration for the previous example:
+
+.. code-block:: Ruby
+
+  http_header_authentication:
+    header: 'sso-user-email'
+    field: 'email'
+    autocreation: false
+
+Autocreation
+------------
+
+Even more, if you want not only *authentication* (authenticating existing users) but also *user creation* you can turn ``autocreation`` on by setting ``autocreation: true``. If you do so, when a user with the trusted header performs his first request his user will be created automatically. This feature requires that ``field`` is set to ``email``, since the new user will be created with it:
+
+* ``email``: value of the header (``alice@myorg.com``).
+* ``username``: user of the email ( ``alice``).
+* ``password``: random. He can change it in his account page.
+* ``organization``: taken from the subdomain (``myorg``).

doc/manual/source/operations/run_sync_tables.rst

@@ -0,0 +1,15 @@
+Running Sync Tables
+===================
+
+If you are working with the **Sync Tables** feature, you must run a rake task to trigger the synchronization of the dataset. This rake retrieves all sync tables that should get synchronized, and puts the synchronization tasks at Resque:
+
+.. highlight:: bash
+
+::
+
+    bundle exec rake cartodb:sync_tables[true]
+
+You might want to set up a cron so that this task is executed periodically in an automated way.
+
+
+

doc/manual/source/run.rst

@@ -0,0 +1,56 @@
+Running CARTO
+=============
+
+First run, setting up an user
+-----------------------------
+
+First run, setting up first time to run your development version of CARTO. Let's suppose that we are going to create a development env and that our user/subdomain is going to be 'development'
+
+.. highlight:: bash
+
+::
+
+    cd cartodb
+    export SUBDOMAIN=development
+
+    # Add entries to /etc/hosts needed in development
+    echo "127.0.0.1 ${SUBDOMAIN}.localhost.lan" | sudo tee -a /etc/hosts
+
+    # Create a development user
+    sh script/create_dev_user
+
+
+Running all the processes
+-------------------------
+
+Start the resque daemon (needed for import jobs):
+
+.. highlight:: bash
+
+::
+
+    bundle exec script/resque
+
+Finally, start the CARTO development server on port 3000:
+
+.. highlight:: bash
+
+::
+
+   bundle exec thin start --threaded -p 3000 --threadpool-size 5
+
+Node apps
+
+.. highlight:: bash
+
+::
+
+    cd cartodb-sql-api && node app.js
+    cd windshaft-cartodb && node app.js
+
+
+You should now be able to access
+**`http://<mysubdomain>.localhost.lan:3000`**
+in your browser and login with the password specified above.
+
+Enjoy