Reorder dev center folder

docs/developer-center/reference/06-isoline-functions.md

@@ -16,6 +16,8 @@ The following functions provide an isoline generator service, based on time or d
 
 Displays a contoured line on a map, connecting geometries to a defined area, measured by an equal range of distance (in meters).
 
+Note that not all the providers, for example TomTom, provide us a way to define the isoline limit in distance so we need to make some estimations. Due that estimations the produced isolines could not be 100% precise.
+
 ##### Arguments
 
 Name | Type | Description | Accepted values

docs/internal/exception_safe.md

@@ -0,0 +1,36 @@
+# Exception-Safe functions
+
+The public API dataservices functions emit exceptions in general when an error occurs
+or a limiting condition is met (e.g. quotas are exceeded).
+
+For each public function `func` we have a internal function named `_func_exception_safe` which
+acts as a wrapper to the public function, with the same signature, but captures
+exceptions generated during its execution (except those due to incomplete configuration or
+authentication issues) and returns NULL or empty set values in those cases.
+
+Please note these functions are considered **not public** and therefore their API (including which exceptions are wrapped and which ones are not) may change.
+
+Instead of raising an exception they raise warnings, hopefully containing the same information of the original exception.
+
+## Intended Use
+
+These functions are useful in cases when it is undesirable to rollback a transaction.
+Fo example if a table is geocoded with:
+
+```sql
+UPDATE table SET the_geom=cdb_geocode_street_point(address,city,NULL,country);
+```
+
+In case of the user geocoding quota being exhausted mid-process, the user could
+incur in external service expenses but any geocoded data would be lost due to the
+transaction rollback.
+
+We can avoid the problem using the corresponding exception-safe function:
+
+```sql
+UPDATE table SET the_geom=_cdb_geocode_street_point_exception_safe(address,city,NULL,country);
+```
+
+# Addition Information
+
+See https://github.com/CartoDB/dataservices-api/issues/314 for more information.

docs/internal/internal_doc.md

@@ -0,0 +1,69 @@
+# Data Services API Internal documentation
+
+* [Existent services](#existent-services)
+* [How to add a new service](#how-to-add-a-new-service)
+
+## Existent services
+
+Available at [cartodb_services](https://github.com/CartoDB/geocoder-api/tree/master/server/lib/python/cartodb_services/cartodb_services).
+
+* Google
+  * [Geocoding](https://github.com/CartoDB/geocoder-api/blob/983440086d3fabf03aedc66f53dcf4c4a8cb2323/server/lib/python/cartodb_services/cartodb_services/google/geocoder.py)
+
+* Here
+  * [Geocoding](https://github.com/CartoDB/geocoder-api/blob/983440086d3fabf03aedc66f53dcf4c4a8cb2323/server/lib/python/cartodb_services/cartodb_services/here/geocoder.py)
+  * [Routing](https://github.com/CartoDB/geocoder-api/blob/983440086d3fabf03aedc66f53dcf4c4a8cb2323/server/lib/python/cartodb_services/cartodb_services/here/routing.py)
+
+## How to add a new service
+
+These are the steps that need to be followed when creating a new service in the API or updating an existent one.
+
+### Creating a new service function or editing an existent one
+In this scenario, both client and server sides require to be edited/created.
+
+* **Update the interface file** with the function addition or update
+  * Interfaces are stored in `client/renderer/interfaces`
+  * Interface YAML filenames follow the client versioning schema with the format `interface-x.y.z.yaml`.
+
+* Update the renderer templates or script, *if applicable*
+  * Renderer templates are stored in `client/renderer/templates`
+  * The Renderer script (`client/renderer/sql-template-renderer`) generates SQL from the defined interfaces
+
+* Generate a **new subfolder version** for `sql` and `test` folders to define the new functions and tests
+  * TODO: Use symlinks to avoid file duplication between versions that don't update them
+  * The `client/sql` folder contents are generated from the interfaces in the first step
+  * Add or upgrade your SQL server functions
+  * Create tests for the client and server functions -- at least, to check that those are created
+
+* Generate the **upgrade and downgrade files** for the extension for both client and server
+
+* Update the control files and the Makefiles to generate the complete SQL file for the new created version
+  * These new version files (`cdb_dataservices_client--X.Y.Z.sql and cdb_dataservices_server--X.Y.X.sql`) must be pushed and frozen. You can add these to the `.gitignore` file.
+
+* Update the public docs! ;-)
+
+### Updating an existing server side function
+
+With no changes in client side.
+
+#### Extension
+
+* Generate a **new subfolder version** for `sql` and `test` folders to define the new functions and tests
+  * **TODO:** Use symlinks to avoid file duplication between versions that don't update them
+  * **Add or upgrade your SQL server functions**
+    * For example, if a new street geocoder service is implemented, it will require a change in the main function (`cdb_dataservices_server.cdb_geocode_street_point`) and generate a new `cdb_dataservices_server._cdb_newservice_geocode_street_point`
+
+* Generate the upgrade and downgrade files for the extension for the client
+
+#### Python
+
+* Add, if needed, [new configuration elements](https://github.com/CartoDB/geocoder-api/blob/983440086d3fabf03aedc66f53dcf4c4a8cb2323/server/lib/python/cartodb_services/cartodb_services/metrics/config.py#L100)
+
+* Add the new **functionality** into the provider folder. If the provider is new, create a new folder(`server/lib/python/cartodb_services/cartodb_services/{provider_name}` and add the service (`geocoder.py`)
+
+* Check the `__init__.py` files to follow the existent [import conventions](https://github.com/CartoDB/geocoder-api/blob/983440086d3fabf03aedc66f53dcf4c4a8cb2323/server/lib/python/cartodb_services/cartodb_services/metrics/__init__.py)
+
+* Add a **new metric**, if needed, at the [corresponding service](https://github.com/CartoDB/geocoder-api/blob/983440086d3fabf03aedc66f53dcf4c4a8cb2323/server/lib/python/cartodb_services/cartodb_services/metrics/quota.py#L37-L60)
+
+* **Update the package version** in [setup.py](https://github.com/CartoDB/geocoder-api/blob/983440086d3fabf03aedc66f53dcf4c4a8cb2323/server/lib/python/cartodb_services/setup.py)
+

docs/internal/quota_management.md

@@ -0,0 +1,66 @@
+All the services from Dataservices API are subject to quota management: check, limit, etc.
+
+There are three main fields in the quota management:
+
+- **Quota**: Number of requests of this kind the user could make, eg. Number of street geocoding requests
+- **Soft limit**: This flag enables the user to surpass their assigned quota. When this flag is activated, there is no quota check so the user could make all the requests that they want. Consequently, the `hard_limit` flag, when enabled, indicates that the user cannot surpass the assigned quota.
+- **Block price**: Price for every 1000 requests
+
+
+All the user/organization quota information is stored in the user metadata in Redis but managed by the CartoDB Rails app through the User/Organization models. Nevertheless you could change/read the Redis information through the following keys:
+
+- `hgetall rails:users:username`
+- `hgetall rails:orgs:orgname`
+
+This whole information is managed by the CartoDB Rails App too so we could make a numer of useful operations in order to know how much quota do you have, how much quota have you spent this month and so on.
+
+###How can I know the current quota, number of uses, etc for a user?###
+
+You could use the following endpoint to know it: https://<username>.cartodb.com/api/v1/users/<user_id>?api_key=<api_key>
+In the result of this endpoint you can see blocks with all the information. Eg:
+
+```
+"geocoding": {
+    "quota": 1000,
+    "block_price": 1500,
+    "monthly_use": 743,
+    "hard_limit": true
+}
+```
+
+### How can I set a new quota for a user###
+
+This operation could be done through the rails console:
+
+- First you have to connect to the rails console: `bundle exec rails c`
+- Once in the console you have to get the target user/organization:
+  - `u = User.find(username: '<username>')`
+  - `o = Organization.find(name: '<orgname>')`
+- After we have the user/organization, we could change the quota or the hard limit flag for the desired service. I'm going to use geocoding as an example but it could be done with all the services:
+
+  ```
+     u.geocoding_quota = 2000
+     [u.soft_geocoding_limit = true|false]
+     u.save
+     ```
+  ```
+     o.geocoding_quota = 2000
+     o.save
+    ```
+- This way the user now has 2000 requests as their current quota
+- We can only change the hard limit flag for users, not for organizations
+
+### What services we could change?###
+
+  The following list numbers all the current services but this is a living list so it could keep growing in the future:
+
+  - Geocoding (street level): `geocoding_quota`, `soft_geocoding_limit`
+  - Isolines: `here_isolines_quota`, `soft_here_isolines_limit`
+  - Data observatory snapshot: `obs_snapshot_quota`, `soft_obs_snapshot_limit`
+  - Data observatory general: `obs_general_quota`, `soft_obs_general_limit`
+
+### How is the quota spent?###
+
+Almost in all the services: geocoding, data observatory snapshot and general the number of spent credits is calculated per request made (either successful or empty request).
+
+In the case of the isolines service, the number of credits is calculated based on the number of isolines generated by the request. Ie. If your query generates 3 isolines for the request, you've spent 3 isolines credits.