Merge pull request #250 from CartoDB/fix/ch175898/gcp-github-actions

fixed gcloud github actions

.github/workflows/main.yml

@@ -0,0 +1,39 @@
+name: data-services PR testing
+on: push
+jobs:
+  data-services:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        pg_version: [10, 12]
+    env:
+        PG_VERSION: ${{ matrix.pg_version }}
+
+    steps:
+      - uses: actions/checkout@v1
+
+      - name: Set up docker registry
+        env:
+          SA_TO_USE: ${{ secrets.GCS }}
+        run: |
+          echo "${SA_TO_USE}" | base64 -d | docker login -u _json_key --password-stdin https://gcr.io
+
+      - name: Pull base image
+        run: docker pull gcr.io/cartodb-on-gcp-ci-testing/cartodb-postgresql-base:${{ matrix.pg_version }}
+
+      - name: Checkout ci tools repository
+        uses: actions/checkout@v2
+        with:
+          repository: CartoDB/ci-tools
+          path: ci-tools
+          token: ${{ secrets.CARTOFANTE_PAT }}
+
+      - name: Copy ci files to root
+        run: cp ci-tools/repos/${{ github.event.repository.name }}/* .
+
+      - name: Start docker-compose services
+        run: docker-compose -f docker-compose.yaml up -d
+
+      - name: Run tests
+        run: docker-compose -f docker-compose.yaml exec -T postgres-server bash -c "cd /data-services/geocoder/extension && make clean all install installcheck"
+        timeout-minutes: 5

NEWS.md

@@ -0,0 +1,4 @@
+0.0.2 (19/02/2018)
+------------------
+
+Compatibility with PostgreSQL 10.

README.md

@@ -1,5 +1,5 @@
-#CartoDB internal geocoder
+#CARTO internal geocoder
 
 *Disclaimer:*
 
-*The administrative boundaries which are used in this service are the ones offered by our data providers, such as Natural Earth Data, Geonames or Quattroshapes, and do not imply the opinion on the part of CartoDB.*
+*The administrative boundaries which are used in this service are the ones offered by our data providers, such as Natural Earth Data, Geonames or Quattroshapes, and do not imply the opinion on the part of CARTO.*

common-data/README.md

@@ -21,7 +21,7 @@ Data library
 
 **Sync table**: `false`
 
-**Source URL**: http://www.naturalearthdata.com/download/50m/physical/ne_50m_rivers_lake_centerlines.zip
+**Source URL**: http://naciscdn.org/naturalearth/50m/physical/ne_50m_rivers_lake_centerlines.zip
 
 #### Ocean
 **Description**: Oceanic region.
@@ -34,7 +34,7 @@ Data library
 
 **Sync table**: `false`
 
-**Source URL**: http://www.naturalearthdata.com/download/50m/physical/ne_50m_ocean.zip
+**Source URL**: http://naciscdn.org/naturalearth/50m/physical/ne_50m_ocean.zip
 
 #### Land
 **Description**: Land region.
@@ -47,7 +47,7 @@ Data library
 
 **Sync table**: `false`
 
-**Source URL**: http://www.naturalearthdata.com/download/50m/physical/ne_50m_land.zip
+**Source URL**: http://naciscdn.org/naturalearth/50m/physical/ne_50m_land.zip
 
 #### Glaciers
 **Description**: Glaciers and recently de-glaciated areas.
@@ -60,7 +60,7 @@ Data library
 
 **Sync table**: `false`
 
-**Source URL**: http://www.naturalearthdata.com/download/50m/physical/ne_50m_glaciated_areas.zip
+**Source URL**: http://naciscdn.org/naturalearth/50m/physical/ne_50m_glaciated_areas.zip
 
 #### Reefs
 **Description**: Coral reefs and atolls.
@@ -73,7 +73,7 @@ Data library
 
 **Sync table**: `false`
 
-**Source URL**: http://www.naturalearthdata.com/download/10m/physical/ne_10m_reefs.zip
+**Source URL**: http://naciscdn.org/naturalearth/10m/physical/ne_10m_reefs.zip
 
 #### Coastlines
 **Description**: Lines around the world coasts.
@@ -86,7 +86,7 @@ Data library
 
 **Sync table**: `false`
 
-**Source URL**: http://www.naturalearthdata.com/download/110m/physical/ne_110m_coastline.zip
+**Source URL**: http://naciscdn.org/naturalearth/110m/physical/ne_110m_coastline.zip
 
 #### Lakes
 **Description**: Natural and artificial lakes.
@@ -99,7 +99,7 @@ Data library
 
 **Sync table**: `false`
 
-**Source URL**: http://www.naturalearthdata.com/download/50m/physical/ne_50m_lakes.zip
+**Source URL**: http://naciscdn.org/naturalearth/50m/physical/ne_50m_lakes.zip
 
 #### Boundaries of the US National Parks 
 **Description**: National Parks Boundaries.
@@ -220,7 +220,7 @@ Data library
 
 **Sync table**: `false`
 
-**Source URL**: http://www.naturalearthdata.com/download/50m/cultural/ne_50m_urban_areas.zip
+**Source URL**: http://naciscdn.org/naturalearth/50m/cultural/ne_50m_urban_areas.zip
 
 #### Populated places
 **Description**: Most populated places.
@@ -233,7 +233,7 @@ Data library
 
 **Sync table**: `false`
 
-**Source URL**: http://www.naturalearthdata.com/download/50m/cultural/ne_50m_populated_places.zip
+**Source URL**: http://naciscdn.org/naturalearth/50m/cultural/ne_50m_populated_places.zip
 
 #### International seaports
 **Description**: Seaports in the world.
@@ -246,7 +246,7 @@ Data library
 
 **Sync table**: `false`
 
-**Source URL**: http://www.naturalearthdata.com/download/10m/cultural/ne_10m_ports.zip
+**Source URL**: http://naciscdn.org/naturalearth/10m/cultural/ne_10m_ports.zip
 
 #### Pakistan districts
 **Description**: Geometries for districts in Pakistan.
@@ -272,7 +272,7 @@ Data library
 
 **Sync table**: `false`
 
-**Source URL**: http://www.naturalearthdata.com/download/10m/cultural/ne_10m_airports.zip
+**Source URL**: http://naciscdn.org/naturalearth/10m/cultural/ne_10m_airports.zip
 
 #### New York: Areas of interest
 **Description**: Areas of interest labels as depicted in New York: A City of Neighborhoods.
@@ -430,7 +430,7 @@ Data library
 
 **Sync table**: `true`
 
-**Source URL**: http://www.naturalearthdata.com/download/10m/cultural/ne_10m_time_zones.zip
+**Source URL**: http://naciscdn.org/naturalearth/10m/cultural/ne_10m_time_zones.zip
 
 #### Local storm reports
 **Description**: Local storm reports last 24 hours, updated every hour.
@@ -623,7 +623,7 @@ Data library
 
 **Sync table**: `false` (File is truncated in GitHub. Sync disabled to avoid errors)
 
-**Source URL**: http://www.naturalearthdata.com/download/10m/cultural/ne_10m_railroads.zip
+**Source URL**: http://naciscdn.org/naturalearth/10m/cultural/ne_10m_railroads.zip
 
 #### Roads
 **Description**: Roads in the world.
@@ -636,7 +636,7 @@ Data library
 
 **Sync table**: `false`
 
-**Source URL**: http://www.naturalearthdata.com/download/10m/cultural/ne_10m_roads.zip
+**Source URL**: http://naciscdn.org/naturalearth/10m/cultural/ne_10m_roads.zip
 
 ***
 
@@ -798,7 +798,7 @@ Data library
 
 **Sync table**: `false`
 
-**Source URL**: http://www.naturalearthdata.com/download/50m/cultural/ne_10m_admin_1_states_provinces.zip
+**Source URL**: http://naciscdn.org/naturalearth/50m/cultural/ne_10m_admin_1_states_provinces.zip
 
 
 #### Autonomous communities of Spain
@@ -886,7 +886,7 @@ Data library
 
 **Sync table**: `false` 
 
-**Source URL**: http://www.naturalearthdata.com/download/50m/cultural/ne_10m_admin_1_states_provinces.zip
+**Source URL**: http://naciscdn.org/naturalearth/50m/cultural/ne_10m_admin_1_states_provinces.zip
 
 #### World borders
 **Description**: World countries borders.
@@ -899,7 +899,7 @@ Data library
 
 **Sync table**: `false` 
 
-**Source URL**: http://www.naturalearthdata.com/download/10m/cultural/ne_100m_admin_0_countries.zip
+**Source URL**: http://naciscdn.org/naturalearth/10m/cultural/ne_100m_admin_0_countries.zip
 
 #### World borders (High definition)
 **Description**: World countries borders (High definition).
@@ -912,7 +912,7 @@ Data library
 
 **Sync table**: `false` 
 
-**Source URL**: http://www.naturalearthdata.com/download/10m/cultural/ne_10m_admin_0_countries.zip
+**Source URL**: http://naciscdn.org/naturalearth/10m/cultural/ne_10m_admin_0_countries.zip
 
 
 #### New York counties
@@ -939,7 +939,7 @@ Data library
 
 **Sync table**: `false` (Custom dataset from NE 50m_admin1)
 
-**Source URL**: http://www.naturalearthdata.com/download/50m/cultural/ne_10m_admin_1_states_provinces.zip
+**Source URL**: http://naciscdn.org/naturalearth/50m/cultural/ne_10m_admin_1_states_provinces.zip
 
 #### USA counties
 **Description**: All USA counties and their information.
@@ -978,7 +978,7 @@ Data library
 
 **Sync table**: `false` 
 
-**Source URL**: http://www.naturalearthdata.com/download/10m/cultural/ne_10m_admin_0_countries.zip
+**Source URL**: http://naciscdn.org/naturalearth/10m/cultural/ne_10m_admin_0_countries.zip
 
 
 #### South American countries
@@ -1034,7 +1034,7 @@ Data library
 
 **Sync table**: `false` 
 
-**Source URL**: http://www.naturalearthdata.com/download/10m/cultural/ne_10m_admin_0_countries.zip
+**Source URL**: http://naciscdn.org/naturalearth/10m/cultural/ne_10m_admin_0_countries.zip
 
 #### Departments of France
 **Description**: Geometries for the 96 departments of France.
@@ -1047,17 +1047,84 @@ Data library
 
 **Sync table**: `false` 
 
+#### Greater Capital City Statistical Area (GCCSA)
+**Description**: Lands administrative divisions of Australia.
+
+**Source**: [Australian Bureau of Statistics](http://www.abs.gov.au/)
+
+**License**: CC BY: Attribution alone
+
+**Table name**: australia_gccsa
+
+**Sync table**: `false` 
+
+**Source URL**:http://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/2033.0.55.0012011?OpenDocument
+
 #### States and territories of Australia
 **Description**: Lands administrative divisions of Australia.
 
-**Source**: [GeoNetwork](http://salbgeonetwork.grid.unep.ch/geonetwork/srv/en/main.home)
+**Source**: [Australian Bureau of Statistics](http://www.abs.gov.au/)
 
-**License**:
+**License**: CC BY: Attribution alone
 
 **Table name**: australia_adm1
 
 **Sync table**: `false` 
 
+**Source URL**: http://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/2033.0.55.0012011?OpenDocument
+
+#### Australia Statistical Area level 4
+**Description**: 106 Regions with population in the range of 100k-500k
+
+**Source**: [Australian Bureau of Statistics](http://www.abs.gov.au/)
+
+**License**: CC BY: Attribution alone
+
+**Table name**: australia_adm2
+
+**Sync table**:`false`
+
+**Source URL**: http://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/2033.0.55.0012011?OpenDocument
+
+#### Australia Statistical Area level 3
+**Description**: 351 Regions with population in the range of 30k-130k
+
+**Source**: [Australian Bureau of Statistics](http://www.abs.gov.au/)
+
+**License**: CC BY: Attribution alone
+
+**Table name**: australia_adm3
+
+**Sync table**:`false`
+
+**Source URL**: http://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/2033.0.55.0012011?OpenDocument
+
+#### Australia Statistical Area level 2
+**Description**: 2214 Regions with population in the range of 3k-25k
+
+**Source**: [Australian Bureau of Statistics](http://www.abs.gov.au/)
+
+**License**: CC BY: Attribution alone
+
+**Table name**: australia_adm4
+
+**Sync table**:`false`
+
+**Source URL**: http://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/2033.0.55.0012011?OpenDocument
+
+#### Australia Statistical Area level 1
+**Description**: 54805 Regions with population in the range of 200-800
+
+**Source**: [Australian Bureau of Statistics](http://www.abs.gov.au/)
+
+**License**: CC BY: Attribution alone
+
+**Table name**: australia_adm5
+
+**Sync table**:`false`
+
+**Source URL**: http://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/2033.0.55.0012011?OpenDocument
+
 #### Ireland counties
 **Description**: Geometries for traditional and administrative counties in Ireland.
 
@@ -1069,7 +1136,20 @@ Data library
 
 **Sync table**: `false` 
 
-**Source URL**: http://www.naturalearthdata.com/download/10m/cultural/ne_10m_admin_1_states_provinces.zip
+**Source URL**: http://naciscdn.org/naturalearth/10m/cultural/ne_10m_admin_1_states_provinces.zip
+
+#### California Protected Areas
+**Description**: Data about lands that are owned outright (“in fee”) and protected for open space purposes in the US state of California.
+
+**Source**: [CA Lands](http://www.calands.org/)
+
+**License**: Public domain
+
+**Table name**: cpad_2015
+
+**Sync table**: `false` 
+
+**Source URL**: http://atlas.ca.gov/casil/planning/Land_Ownership/GreenInfoNetworkProject/CPAD-2015b-December2015/CPAD_2015b.zip
 
 ***
 

geocoder/.gitignore

@@ -1 +1,2 @@
 db_dumps/
+data_patches/

geocoder/CONTRIBUTING.md

@@ -33,9 +33,16 @@ There may be different kind of changes in the dataset:
 
   Remember to make a backup of the current data running in your production environment before you do any changes.
 
-  Any edition of the data must be performed via SQL queries in order to be run in the build scripts for each section. In order to build the production table, please, edit the build SQL functions as needed, and create a new table in production by running them.
+  Any edition of the data must be performed via SQL queries in order to be run over the existing data to patch it. An example of patch could be:
+  ```sql
+  -- Patch 0.0.1 data version -20160203
 
-  This method will allow us to make sure that the data that we have in a production environment is totally reproducible just by running the setup scripts.
+  DELETE FROM admin0_synonyms WHERE rank = 8 AND char_length(name_) < 4;
+  ```
+  
+  Please, send your patch file in a PR for us to be able to set it up in the patch downloader and loader scripts of the project. Patchs should be associated to a data version and will be stored in: `geocoding/dumps/$VERSION/patches`
+  
+  This method will allow us to make sure that the data that we have in a production environment is totally reproducible just by running the setup scripts and the patches.
 
   **Changes regarding functions:**
 

geocoder/README.md

@@ -1,6 +1,7 @@
-CartoDB Internal Geocoder
+CARTO Internal Geocoder
 ====
 
+
 # API Response Format
 
 ## Parameters
@@ -43,7 +44,7 @@ SELECT (geocode_admin0_polygons(Array['Spain', 'USA', ''])).*
 
 #### Links
 
-[See it on the api](http://geocoding.cartodb.com/api/v2/sql?q=SELECT%20(geocode_admin0_polygons(Array[%27Spain%27,%20%27USA%27,%20%27%27])).*)
+[See it on the api](http://geocoding.carto.com/api/v2/sql?q=SELECT%20(geocode_admin0_polygons(Array[%27Spain%27,%20%27USA%27,%20%27%27])).*)
 
 [See plpgsql](https://github.com/CartoDB/data-services/blob/master/geocoder/admin0/sql/geocoder.sql)
 
@@ -80,7 +81,7 @@ SELECT (geocode_admin1_polygons(Array['az', 'az'], Array['Ecuador', 'USA'])).*
 
 #### Links
 
-[See it on the api](http://geocoding.cartodb.com/api/v2/sql?q=SELECT%20(geocode_admin1_polygons(Array[%27az%27,%20%27az%27],%20Array[%27Ecuador%27,%20%27USA%27])).*)
+[See it on the api](http://geocoding.carto.com/api/v2/sql?q=SELECT%20(geocode_admin1_polygons(Array[%27az%27,%20%27az%27],%20Array[%27Ecuador%27,%20%27USA%27])).*)
 
 [See plpgsql](https://github.com/CartoDB/data-services/blob/master/geocoder/admin1/sql/geocoder.sql)
 
@@ -131,7 +132,7 @@ SELECT (geocode_namedplace(
 
 #### Links
 
-[See it on the api](http://geocoding.cartodb.com/api/v2/sql?q=SELECT%20(geocode_namedplace(Array[%27sunapee%27,%20%27sunapeeee%27,%20%27New%20York%20City%27,%20%27Madrid%27],%20Array[%27%27,%20%27US%27,%20%27United%20States%27,%20NULL])).*)
+[See it on the api](http://geocoding.carto.com/api/v2/sql?q=SELECT%20(geocode_namedplace(Array[%27sunapee%27,%20%27sunapeeee%27,%20%27New%20York%20City%27,%20%27Madrid%27],%20Array[%27%27,%20%27US%27,%20%27United%20States%27,%20NULL])).*)
 
 [See plpgsql](https://github.com/CartoDB/data-services/blob/master/geocoder/namedplace/sql/geocode_namedplace.sql)
 
@@ -158,7 +159,7 @@ SELECT (geocode_ip(Array['1.0.16.0', '::ffff:1.0.16.0', 'not an IP'])).*
 
 #### Links
 
-[See it on the api](http://geocoding.cartodb.com/api/v2/sql?q=SELECT%20(geocode_ip(Array[%271.0.16.0%27,%20%27::ffff:1.0.16.0%27,%20%27not%20an%20IP%27])).*)
+[See it on the api](http://geocoding.carto.com/api/v2/sql?q=SELECT%20(geocode_ip(Array[%271.0.16.0%27,%20%27::ffff:1.0.16.0%27,%20%27not%20an%20IP%27])).*)
 
 [See plpgsql](https://github.com/CartoDB/data-services/blob/master/geocoder/ip-addresses/sql/geocoder.sql)
 
@@ -192,7 +193,7 @@ SELECT (geocode_postalcode_polygons(Array['10013','G9H','03782'], Array['USA', '
 
 #### Links
 
-[See it on the api](http://geocoding.cartodb.com/api/v2/sql?q=SELECT%20(geocode_postalcode_points(Array[%2703204%27],Array[%27ESP%27])).*)
+[See it on the api](http://geocoding.carto.com/api/v2/sql?q=SELECT%20(geocode_postalcode_points(Array[%2703204%27],Array[%27ESP%27])).*)
 
 [See plpgsql](https://github.com/CartoDB/data-services/blob/master/geocoder/postal-codes/sql/geocoder.sql)
 
@@ -226,6 +227,6 @@ SELECT (geocode_postalcode_points(Array['10013','G9H','03782'], Array['USA', 'Ca
 
 #### Links
 
-[See it on the api](https://geocoding.cartodb.com/api/v2/sql?q=SELECT%20(geocode_postalcode_points(Array[%2710013%27,%27G9H%27,%2703782%27],%20Array[%27USA%27,%20%27Canada%27,%20%27US%27])).*)
+[See it on the api](https://geocoding.carto.com/api/v2/sql?q=SELECT%20(geocode_postalcode_points(Array[%2710013%27,%27G9H%27,%2703782%27],%20Array[%27USA%27,%20%27Canada%27,%20%27US%27])).*)
 
 [See plpgsql](https://github.com/CartoDB/data-services/blob/master/geocoder/postal-codes/sql/geocoder.sql)

geocoder/admin0/README.md

@@ -20,7 +20,7 @@ SELECT (geocode_admin0_polygons(Array['Spain', 'USA', ''])).*;
 5. Upload the `data/wikipedia_iso_3166_2` table if it doesn't already exist
 6. Run the `sql/subdivide_polygons.sql`
 7. Run the `sql/build_synonym_table.sql`
-8. If needed, load or replace the function with `sql/geocoder.sql`. Don't forget to create the types: `geocode_admin_v1` and `synonym_lookup_v1`
+8. Create the types: `geocode_admin_v1` and `synonym_lookup_v1`
 
 # Tables
 #### admin0_synonyms
@@ -186,7 +186,7 @@ Indexes:
 * geocode_admin0_polygons
 
 ```
- Schema |          Name           |    Result data type    | Argument data types |  Type  
+ Schema |          Name           |    Result data type    | Argument data types |  Type
 --------+-------------------------+------------------------+---------------------+--------
  public | geocode_admin0_polygons | SETOF geocode_admin_v1 | name text[]         | normal
 ```
@@ -194,7 +194,7 @@ Indexes:
 * admin0_synonym_lookup - Related with admin0 synonym service, see below
 
 ````
- Schema |         Name          |    Result data type     | Argument data types |  Type  
+ Schema |         Name          |    Result data type     | Argument data types |  Type
 --------+-----------------------+-------------------------+---------------------+--------
  public | admin0_synonym_lookup | SETOF synonym_lookup_v1 | name text[]         | normal
 ````
@@ -202,7 +202,7 @@ Indexes:
 * [admin0_available_services](https://github.com/CartoDB/data-services/blob/master/geocoder/available-services/sql/services.sql) - available in `geocoder/available-services`
 
 ````
- Schema |           Name            |      Result data type       | Argument data types |  Type  
+ Schema |           Name            |      Result data type       | Argument data types |  Type
 --------+---------------------------+-----------------------------+---------------------+--------
  public | admin0_available_services | SETOF available_services_v1 | name text[]         | normal
 ````
@@ -214,7 +214,7 @@ Indexes:
 
 # Data Sources
 
-- **Natural Earth Data**: ne_10m_admin_0_countries (version 3.0) which is currently stored in Geocoding.CartoDB as ne_admin0_v3. This is stored in data/ne_10m_admin_0_countries.zip
+- **Natural Earth Data**: ne_10m_admin_0_countries (version 3.0) which is currently stored in `geocoding.carto.com` as ne_admin0_v3. This is stored in data/ne_10m_admin_0_countries.zip
   - Coverage: global
   - Geometry type: polygon
 
@@ -290,6 +290,10 @@ For country boundaries, we use the data provided by Natural Earth Data. The whol
 # Known issues
 
 # Historic
+* [04/10/2017]:
+  * 0.0.3: Merge Corsica with France
+* [01/12/2015]:
+  * Removed geocoder functions. Check the /extension folder instead.
 * [19/10/2015]:
   * Revision and usage examples addition
 * [06/10/2015]:

geocoder/admin0/data/admin0_synonym_additions.csv

@@ -14,6 +14,7 @@ COD,"Congo, Dem. Rep.",data source: world bank,10
 COD,Democratic republic of the Congo,,10
 COD, The Democratic Republic of the Congo,,10
 COG,The Republic of the Congo,,10
+COG,Republic of the Congo,,10
 COG,"Congo, Rep.",data source: world bank,10
 COG,Congo (CG),,10
 EGY,"Egypt, Arab Rep.",data source: world bank,10

geocoder/admin0/patches/20160203_countries_bh_isocode.sql

@@ -0,0 +1,3 @@
+-- Patch 0.0.1 data version -20160203
+
+DELETE FROM admin0_synonyms WHERE rank = 8 AND char_length(name_) < 4;

geocoder/admin0/patches/20160622_countries_synonym_congo.sql

@@ -0,0 +1,9 @@
+-- Patch 0.0.1 data version -20160622
+DO $$
+    DECLARE exiting INTEGER;
+BEGIN
+    SELECT count(*) FROM admin0_synonyms WHERE name = 'Republic of the Congo' AND adm0_a3 = 'COG' and rank = 10 INTO exiting;
+    IF exiting = 0 THEN
+        INSERT INTO admin0_synonyms (name, rank, adm0_a3) VALUES ('Republic of the Congo', 10, 'COG');
+    END IF;
+END$$;

geocoder/admin0/patches/20171004_merge_corsica_and_france.sql

@@ -0,0 +1,2 @@
+UPDATE ne_admin0_v3 SET the_geom = (SELECT ST_Union(the_geom) FROM ne_admin0_v3 WHERE adm0_a3 IN ('FRA', 'FRH')) WHERE adm0_a3 = 'FRA';
+DELETE FROM ne_admin0_v3 WHERE adm0_a3 = 'FRH';

geocoder/admin0/patches/20180306_add_ssd_rows_for_south_sudan.sql

@@ -0,0 +1,8 @@
+DO $$
+  DECLARE updated INTEGER;
+BEGIN
+  SELECT count(*) FROM admin0_synonyms WHERE adm0_a3 = 'SSD' INTO updated;
+  IF updated = 0 THEN
+    INSERT INTO admin0_synonyms (name, rank, adm0_a3, name_) SELECT name, rank, 'SSD' AS adm0_a3, name_ FROM admin0_synonyms WHERE adm0_a3 = 'SDS';
+  END IF;
+END$$;

geocoder/admin0/patches/20181011_add_synonyms_for_swaziland.sql

@@ -0,0 +1,9 @@
+DO $$
+  DECLARE updated INTEGER;
+BEGIN
+  SELECT count(*) FROM admin0_synonyms WHERE adm0_a3 = 'SWZ' GROUP BY adm0_a3 HAVING count(*) > 0 INTO updated;
+  IF updated = 4 THEN
+    INSERT INTO admin0_synonyms (name, rank, adm0_a3, name_) SELECT 'Eswatini' as name, 6, adm0_a3, 'eswatini' as name_ FROM admin0_synonyms WHERE name_ = 'swaziland';
+    INSERT INTO admin0_synonyms (name, rank, adm0_a3, name_) SELECT 'Kingdom of Eswatini' as name, 7, adm0_a3, 'kingdomofeswatini' as name_ FROM admin0_synonyms WHERE name_ = 'swaziland';
+  END IF;
+END$$;

geocoder/admin0/patches/20210118_add_renamed_country_north_macedonia.sql

@@ -0,0 +1,9 @@
+DO $$
+  DECLARE updated INTEGER;
+BEGIN
+  SELECT count(*) FROM admin0_synonyms WHERE adm0_a3 = 'MKD' GROUP BY adm0_a3 HAVING count(*) > 0 INTO updated;
+  IF updated = 6 THEN
+    INSERT INTO admin0_synonyms (name, rank, adm0_a3, name_) SELECT 'North Macedonia' as name, 6, adm0_a3, 'northmacedonia' as name_ FROM admin0_synonyms WHERE name_ = 'macedonia';
+    INSERT INTO admin0_synonyms (name, rank, adm0_a3, name_) SELECT 'Republic of North Macedonia' as name, 7, adm0_a3, 'republicofnorthmacedonia' as name_ FROM admin0_synonyms WHERE name_ = 'macedonia';
+  END IF;
+END$$;

geocoder/admin0/sql/build_synonym_table.sql

@@ -81,7 +81,9 @@ INSERT INTO admin0_synonyms (name, rank,  adm0_a3)
 SELECT 
     abbrev, 8, adm0_a3
 FROM
-    ne_admin0_v3;
+    ne_admin0_v3
+WHERE
+    char_length(regexp_replace(abbrev, '[^a-zA-Z\u00C0-\u00ff]+', '', 'g')) > 3;
 
 -- insert subunit as name with a rank = 9    
 INSERT INTO admin0_synonyms (name, rank, adm0_a3)

geocoder/admin0/sql/subdivide_polygons.sql

@@ -7,13 +7,6 @@ WITH a AS (SELECT (ST_Dump(the_geom)).geom geom,adm0_a3  FROM ne_admin0_v3 WHERE
 
 SELECT ST_Collect(geom), 'GUF', 'French Guiane'  FROM a WHERE ST_Intersects(geom, ST_Buffer(CDB_LatLNg(4, -53), 8));
 
--- Split Corse from France
-INSERT INTO ne_admin0_v3 (the_geom, adm0_a3, name)
-
-WITH a AS (SELECT (ST_Dump(the_geom)).geom geom,adm0_a3  FROM ne_admin0_v3 WHERE ST_Intersects(the_geom, CDB_LatLNg(42.14, 9.12)))
-
-SELECT ST_Collect(geom), 'FRH', 'Corse'  FROM a WHERE ST_Intersects(geom, ST_Buffer(CDB_LatLNg(42.14, 9.12), 2));
-
 -- Split Reunion from France
 INSERT INTO ne_admin0_v3 (the_geom, adm0_a3, name)
 
@@ -23,7 +16,7 @@ SELECT ST_Collect(geom), 'REU', 'Reunion'  FROM a WHERE ST_Intersects(geom, ST_B
 
 -- Remove the above three from the FRA polygon
 WITH a AS (SELECT (ST_Dump(the_geom)).geom geom  FROM ne_admin0_v3 WHERE adm0_a3 = 'FRA')
-UPDATE ne_admin0_v3 SET the_geom = (SELECT ST_Union(geom) FROM a WHERE NOT ST_intersects(geom, (SELECT ST_Union(the_geom) FROM ne_admin0_v3 WHERE adm0_a3 IN ('GUF', 'FRH', 'REU')))) WHERE adm0_a3 = 'FRA';
+UPDATE ne_admin0_v3 SET the_geom = (SELECT ST_Union(geom) FROM a WHERE NOT ST_intersects(geom, (SELECT ST_Union(the_geom) FROM ne_admin0_v3 WHERE adm0_a3 IN ('GUF', 'REU')))) WHERE adm0_a3 = 'FRA';
 
 -- Split Martinique from France
 

geocoder/admin0/test/data/test.sh

@@ -6,10 +6,10 @@
 function test_geocoding_quality_admin0() {
     # checks that the number of geometries is the expected - includes islands and regions splitted in the setup
     # scripts, and some geometries for disputed territories
-    sql 'SELECT count(*) FROM ne_admin0_v3' should 267
+    sql 'SELECT count(*) FROM ne_admin0_v3' should 266
 
     # checks the type of the geometries
-    sql "SELECT ST_GeometryType(the_geom) FROM ne_admin0_v3" should ST_MultiPolygon
+    sql "SELECT ST_GeometryType(the_geom) FROM ne_admin0_v3 GROUP BY 1" should ST_MultiPolygon
 
     # checks that the synonym table includes at least two rows per region: ISO2 code and name
     sql "WITH q AS (SELECT adm0_a3 FROM admin0_synonyms group by adm0_a3 having count(*) < 2) SELECT count(*) FROM q" should 0
@@ -98,7 +98,6 @@ function test_geocoding_quality_admin0() {
     sql "SELECT ST_Intersects(ST_GeomFromText('POLYGON((-180 -21.7111141909999,-180 -12.4752743469999,180 -12.4752743469999,180 -21.7111141909999,-180 -21.7111141909999))', 4326), ST_Centroid(the_geom)) FROM ne_admin0_v3 where adm0_a3 = 'FJI'" should true
     sql "SELECT ST_Intersects(ST_GeomFromText('POLYGON((-61.3181860019999 -52.4065227259999,-61.3181860019999 -51.0277645809999,-57.7342830069999 -51.0277645809999,-57.7342830069999 -52.4065227259999,-61.3181860019999 -52.4065227259999))', 4326), ST_Centroid(the_geom)) FROM ne_admin0_v3 where adm0_a3 = 'FLK'" should true
     sql "SELECT ST_Intersects(ST_GeomFromText('POLYGON((-54.6152921149999 2.11067332000013,-54.6152921149999 51.0875408834804,8.20030521600006 51.0875408834804,8.20030521600006 2.11067332000013,-54.6152921149999 2.11067332000013))', 4326), ST_Centroid(the_geom)) FROM ne_admin0_v3 where adm0_a3 = 'FRA'" should true
-    sql "SELECT ST_Intersects(ST_GeomFromText('POLYGON((8.54525800900012 41.365912177,8.54525800900012 43.0174014340001,9.55958092500009 43.0174014340001,9.55958092500009 41.365912177,8.54525800900012 41.365912177))', 4326), ST_Centroid(the_geom)) FROM ne_admin0_v3 where adm0_a3 = 'FRH'" should true
     sql "SELECT ST_Intersects(ST_GeomFromText('POLYGON((-7.64415442599994 61.3941104190001,-7.64415442599994 62.3989118510001,-6.2757869129999 62.3989118510001,-6.2757869129999 61.3941104190001,-7.64415442599994 61.3941104190001))', 4326), ST_Centroid(the_geom)) FROM ne_admin0_v3 where adm0_a3 = 'FRO'" should true
     sql "SELECT ST_Intersects(ST_GeomFromText('POLYGON((138.063812696 0.918158270000148,138.063812696 9.77558014500013,163.046560092 9.77558014500013,163.046560092 0.918158270000148,138.063812696 0.918158270000148))', 4326), ST_Centroid(the_geom)) FROM ne_admin0_v3 where adm0_a3 = 'FSM'" should true
     sql "SELECT ST_Intersects(ST_GeomFromText('POLYGON((8.69556725400014 -3.93685618990304,8.69556725400014 2.32249501600009,14.4989905190001 2.32249501600009,14.4989905190001 -3.93685618990304,8.69556725400014 -3.93685618990304))', 4326), ST_Centroid(the_geom)) FROM ne_admin0_v3 where adm0_a3 = 'GAB'" should true

geocoder/admin0/test/functions/test.sh

@@ -10,9 +10,9 @@ function test_geocoding_functions_admin0() {
     sql "SELECT (admin0_synonym_lookup(Array['Null Island'])).adm0_a3 is null" should true
 
     # checks that all the geometries have the expected type: ST_MultiPolygon
-    sql "select distinct(st_geometrytype((geocode_admin0_polygons(Array['AGO', 'FRH', 'REU', 'BHR', 'BHS', 'BLR', 'CHN', 'CSI', 'COL', 'KOR', 'AFG', 'ATC', 'ATG', 'AUT', 'VUT', 'SXM', 'USA', 'UZB', 'LAO', 'MAF', 'MAR', 'MOZ', 'ROU', 'SDN', 'SDS', 'SOM', 'SYR', 'URY', 'ABW', 'AUS', 'AIA', 'ALB', 'BEN', 'ARG', 'ATA', 'AZE', 'BIH', 'BJN', 'ARE', 'ALD', 'AND', 'ARM', 'ATF', 'BGR', 'PAK', 'BLM', 'BLZ', 'CUW', 'BMU', 'BOL', 'BDI', 'BEL', 'BFA', 'BGD', 'BRA', 'BRB', 'CHE', 'CHL', 'CIV', 'IDN', 'OMN', 'COG', 'HUN', 'IRQ', 'NOR', 'BRN', 'CLP', 'CMR', 'COD', 'COK', 'GIB', 'GIN', 'NPL', 'FRA', 'CNM', 'BTN', 'BWA', 'CAF', 'CAN', 'COM', 'CYM', 'CPV', 'CRI', 'CUB', 'ECU', 'ISL', 'CYN', 'EGY', 'CYP', 'CZE', 'DEU', 'ERI', 'DJI', 'TGO', 'DMA', 'DNK', 'DOM', 'DZA', 'GUM', 'GUY', 'ESB', 'ESP', 'EST', 'ETH', 'FIN', 'FJI', 'FLK', 'FRO', 'KGZ', 'GRC', 'NRU', 'FSM', 'GAB', 'GBR', 'GEO', 'KHM', 'KIR', 'NCL', 'GGY', 'GHA', 'GNQ', 'GMB', 'GNB', 'GRD', 'GRL', 'HKG', 'GTM', 'HMD', 'HND', 'HRV', 'HTI', 'IMN', 'IND', 'IOT', 'IRL', 'IRN', 'KAB', 'KAS', 'KEN', 'NIU', 'NER', 'KAZ', 'JAM', 'JEY', 'MDV', 'ISR', 'ITA', 'JOR', 'JPN', 'MEX', 'KNA', 'KOS', 'MMR', 'LCA', 'LIE', 'MAC', 'NLD', 'KWT', 'LKA', 'MLT', 'LBN', 'LBR', 'LBY', 'MCO', 'LSO', 'LTU', 'MDA', 'MDG', 'LUX', 'LVA', 'MHL', 'RUS', 'MNP', 'NZL', 'MKD', 'MLI', 'MRT', 'MNE', 'MNG', 'THA', 'MSR', 'MUS', 'MWI', 'MYS', 'NAM', 'NFK', 'NGA', 'NIC', 'PAN', 'PCN', 'PER', 'TJK', 'PGA', 'PHL', 'SLE', 'PRK', 'WSB', 'SHN', 'SLB', 'SPM', 'PLW', 'PNG', 'POL', 'PRI', 'PRT', 'PRY', 'SEN', 'PSX', 'PYF', 'QAT', 'SCR', 'STP', 'SUR', 'SVK', 'SVN', 'SWE', 'SRB', 'UKR', 'RWA', 'SER', 'VAT', 'SGP', 'SAH', 'SAU', 'SGS', 'UGA', 'SOL', 'TUR', 'WLF', 'SWZ', 'SLV', 'SMR', 'TCA', 'TCD', 'SYC', 'TKM', 'YEM', 'TLS', 'TUV', 'ZAF', 'VCT', 'VEN', 'TON', 'TTO', 'TUN', 'TWN', 'TZA', 'UMI', 'USG', 'VGB', 'VIR', 'VNM', 'WSM', 'ZMB', 'ZWE', 'CXR', 'MTQ', 'MYT', 'GLP', 'SJM', 'CCK', 'BES', 'TKL', 'ASM', 'IOA', 'BVT', 'GUF'])).geom))" should ST_MultiPolygon
- 
-    # checks that the synonym service includes the official english name of the regions 
+    sql "select distinct(st_geometrytype((geocode_admin0_polygons(Array['AGO', 'REU', 'BHR', 'BHS', 'BLR', 'CHN', 'CSI', 'COL', 'KOR', 'AFG', 'ATC', 'ATG', 'AUT', 'VUT', 'SXM', 'USA', 'UZB', 'LAO', 'MAF', 'MAR', 'MOZ', 'ROU', 'SDN', 'SDS', 'SOM', 'SYR', 'URY', 'ABW', 'AUS', 'AIA', 'ALB', 'BEN', 'ARG', 'ATA', 'AZE', 'BIH', 'BJN', 'ARE', 'ALD', 'AND', 'ARM', 'ATF', 'BGR', 'PAK', 'BLM', 'BLZ', 'CUW', 'BMU', 'BOL', 'BDI', 'BEL', 'BFA', 'BGD', 'BRA', 'BRB', 'CHE', 'CHL', 'CIV', 'IDN', 'OMN', 'COG', 'HUN', 'IRQ', 'NOR', 'BRN', 'CLP', 'CMR', 'COD', 'COK', 'GIB', 'GIN', 'NPL', 'FRA', 'CNM', 'BTN', 'BWA', 'CAF', 'CAN', 'COM', 'CYM', 'CPV', 'CRI', 'CUB', 'ECU', 'ISL', 'CYN', 'EGY', 'CYP', 'CZE', 'DEU', 'ERI', 'DJI', 'TGO', 'DMA', 'DNK', 'DOM', 'DZA', 'GUM', 'GUY', 'ESB', 'ESP', 'EST', 'ETH', 'FIN', 'FJI', 'FLK', 'FRO', 'KGZ', 'GRC', 'NRU', 'FSM', 'GAB', 'GBR', 'GEO', 'KHM', 'KIR', 'NCL', 'GGY', 'GHA', 'GNQ', 'GMB', 'GNB', 'GRD', 'GRL', 'HKG', 'GTM', 'HMD', 'HND', 'HRV', 'HTI', 'IMN', 'IND', 'IOT', 'IRL', 'IRN', 'KAB', 'KAS', 'KEN', 'NIU', 'NER', 'KAZ', 'JAM', 'JEY', 'MDV', 'ISR', 'ITA', 'JOR', 'JPN', 'MEX', 'KNA', 'KOS', 'MMR', 'LCA', 'LIE', 'MAC', 'NLD', 'KWT', 'LKA', 'MLT', 'LBN', 'LBR', 'LBY', 'MCO', 'LSO', 'LTU', 'MDA', 'MDG', 'LUX', 'LVA', 'MHL', 'RUS', 'MNP', 'NZL', 'MKD', 'MLI', 'MRT', 'MNE', 'MNG', 'THA', 'MSR', 'MUS', 'MWI', 'MYS', 'NAM', 'NFK', 'NGA', 'NIC', 'PAN', 'PCN', 'PER', 'TJK', 'PGA', 'PHL', 'SLE', 'PRK', 'WSB', 'SHN', 'SLB', 'SPM', 'PLW', 'PNG', 'POL', 'PRI', 'PRT', 'PRY', 'SEN', 'PSX', 'PYF', 'QAT', 'SCR', 'STP', 'SUR', 'SVK', 'SVN', 'SWE', 'SRB', 'UKR', 'RWA', 'SER', 'VAT', 'SGP', 'SAH', 'SAU', 'SGS', 'UGA', 'SOL', 'TUR', 'WLF', 'SWZ', 'SLV', 'SMR', 'TCA', 'TCD', 'SYC', 'TKM', 'YEM', 'TLS', 'TUV', 'ZAF', 'VCT', 'VEN', 'TON', 'TTO', 'TUN', 'TWN', 'TZA', 'UMI', 'USG', 'VGB', 'VIR', 'VNM', 'WSM', 'ZMB', 'ZWE', 'CXR', 'MTQ', 'MYT', 'GLP', 'SJM', 'CCK', 'BES', 'TKL', 'ASM', 'IOA', 'BVT', 'GUF'])).geom))" should ST_MultiPolygon
+
+    # checks that the synonym service includes the official english name of the regions
     sql "SELECT (admin0_synonym_lookup(Array['Azerbaijan'])).adm0_a3" should AZE
     sql "SELECT (admin0_synonym_lookup(Array['Georgia'])).adm0_a3" should GEO
     sql "SELECT (admin0_synonym_lookup(Array['Bahrain'])).adm0_a3" should BHR
@@ -352,7 +352,6 @@ function test_geocoding_functions_admin0() {
     sql "SELECT ST_Intersects(ST_GeomFromText('POLYGON((-180 -21.7111141909999,-180 -12.4752743469999,180 -12.4752743469999,180 -21.7111141909999,-180 -21.7111141909999))', 4326), ST_Centroid((geocode_admin0_polygons(Array['FJI'])).geom))" should true
     sql "SELECT ST_Intersects(ST_GeomFromText('POLYGON((-61.3181860019999 -52.4065227259999,-61.3181860019999 -51.0277645809999,-57.7342830069999 -51.0277645809999,-57.7342830069999 -52.4065227259999,-61.3181860019999 -52.4065227259999))', 4326), ST_Centroid((geocode_admin0_polygons(Array['FLK'])).geom))" should true
     sql "SELECT ST_Intersects(ST_GeomFromText('POLYGON((-54.6152921149999 2.11067332000013,-54.6152921149999 51.0875408834804,8.20030521600006 51.0875408834804,8.20030521600006 2.11067332000013,-54.6152921149999 2.11067332000013))', 4326), ST_Centroid((geocode_admin0_polygons(Array['FRA'])).geom))" should true
-    sql "SELECT ST_Intersects(ST_GeomFromText('POLYGON((8.54525800900012 41.365912177,8.54525800900012 43.0174014340001,9.55958092500009 43.0174014340001,9.55958092500009 41.365912177,8.54525800900012 41.365912177))', 4326), ST_Centroid((geocode_admin0_polygons(Array['FRH'])).geom))" should true
     sql "SELECT ST_Intersects(ST_GeomFromText('POLYGON((-7.64415442599994 61.3941104190001,-7.64415442599994 62.3989118510001,-6.2757869129999 62.3989118510001,-6.2757869129999 61.3941104190001,-7.64415442599994 61.3941104190001))', 4326), ST_Centroid((geocode_admin0_polygons(Array['FRO'])).geom))" should true
     sql "SELECT ST_Intersects(ST_GeomFromText('POLYGON((138.063812696 0.918158270000148,138.063812696 9.77558014500013,163.046560092 9.77558014500013,163.046560092 0.918158270000148,138.063812696 0.918158270000148))', 4326), ST_Centroid((geocode_admin0_polygons(Array['FSM'])).geom))" should true
     sql "SELECT ST_Intersects(ST_GeomFromText('POLYGON((8.69556725400014 -3.93685618990304,8.69556725400014 2.32249501600009,14.4989905190001 2.32249501600009,14.4989905190001 -3.93685618990304,8.69556725400014 -3.93685618990304))', 4326), ST_Centroid((geocode_admin0_polygons(Array['GAB'])).geom))" should true

geocoder/admin1/README.md

@@ -19,7 +19,6 @@ SELECT (geocode_admin1_polygons(Array['Alicante', 'California'], Array['Spain',
 3. If fresh, add all `sql/indexes.sql` and `sql/triggers.sql`
 4. Run the `sql/build_data_table.sql` script.
 5. Run the `sql/build_admin1_synonyms.sql` script.
-6. If needed, load or replace the function with `sql/geocoder.sql`
 
 # Tables
 
@@ -206,10 +205,12 @@ The table contains the following columns to be populated:
 
 # Known issues
 * `admin1_decoder` table which is meant to be depreacted is being used in other geocoders, as namedplaces
-* All the information in this README.md doesn't contain the actual status of the CartoDB geocoder, which is using the table `global_province_polygons` documented in the deprecated ADM1 geocoder section.
+* All the information in this README.md doesn't contain the actual status of the CARTO geocoder, which is using the table `global_province_polygons` documented in the deprecated ADM1 geocoder section.
 * In Italy, provinces are being shown instead of ADMIN1 regions. The same happened with Spain, which is manually fixed.
 
 # Historic:
+* [01/12/2015]:
+  * Removed geocoder function. Check /extensions instead. 
 * [19/10/2015]:
   * Updates on README + Adding usage examples
 * [08/10/2015]:

geocoder/admin1/patches/20180117_hsinchu_synonyms.sql

@@ -0,0 +1,4 @@
+UPDATE global_province_polygons
+  SET synonyms = array_append(array_remove(synonyms, 'hsinchu'), 'hsinchu county'),
+      "name" = 'Hsinchu County'
+  WHERE adm1_code = 'TWN-1162';

geocoder/admin1/patches/20190111_france_regions_typos.sql

@@ -0,0 +1,9 @@
+UPDATE global_province_polygons
+  SET synonyms = array_append(array_remove(synonyms, 'meurhe-et-moselle'), 'meurthe-et-moselle'),
+      "name" = 'Meurthe-et-Moselle'
+  WHERE adm1_code = 'FRA-5325';
+
+UPDATE global_province_polygons
+  SET synonyms = array_append(array_remove(synonyms, 'seien-et-marne'), 'seine-et-marne'),
+      "name" = 'Seine-et-Marne'
+  WHERE adm1_code = 'FRA-5342';

geocoder/admin1/patches/20220325_france_region_haut-rhin_typo.sql

@@ -0,0 +1,3 @@
+UPDATE global_province_polygons
+  SET synonyms = array_append(array_remove(synonyms, 'haut-rhin'), 'haut-rhin')
+  WHERE adm1_code = 'FRA-5296';

geocoder/carto-package.json

@@ -0,0 +1,11 @@
+{
+  "name": "dataservices-geocoder-extension",
+  "current_version": {
+    "requires": {
+      "postgresql": "^10.0.0",
+      "postgis": "^2.4.0.0",
+      "carto_postgresql_ext": "^0.23.0"
+    }
+  }
+}
+

geocoder/extension/Makefile

@@ -3,24 +3,28 @@
 EXTENSION = cdb_geocoder
 EXTVERSION = $(shell grep default_version $(EXTENSION).control | sed -e "s/default_version[[:space:]]*=[[:space:]]*'\([^']*\)'/\1/")
 
-DATA = $(EXTENSION)--$(EXTVERSION).sql
+NEW_EXTENSION_ARTIFACT = $(EXTENSION)--$(EXTVERSION).sql
+DATA = $(NEW_EXTENSION_ARTIFACT)
 
-REGRESS = $(notdir $(basename $(wildcard sql/*test.sql)))
+REGRESS = $(notdir $(basename $(sort $(wildcard test/sql/*test.sql))))
+TEST_DIR = test/
+REGRESS_OPTS = --inputdir='$(TEST_DIR)' --outputdir='$(TEST_DIR)' --user='postgres'
 
 # postgres build stuff
 PG_CONFIG = pg_config
 PGXS := $(shell $(PG_CONFIG) --pgxs)
 include $(PGXS)
 
+SOURCES_DATA = $(wildcard sql/*.sql)
 
-SOURCES_DATA = $(wildcard sql/$(EXTVERSION)/*.sql)
-
-$(DATA): $(SOURCES_DATA)
+$(NEW_EXTENSION_ARTIFACT): $(SOURCES_DATA)
 	rm -f $@
 	cat $(SOURCES_DATA) >> $@
 
+.PHONY: all
 all: $(DATA)
 
 # Only meant for development time, do not use once a version is released
+.PHONY: devclean
 devclean:
 	rm -f $(DATA)

geocoder/extension/README.md

@@ -1,23 +1,23 @@
-# CartoDB geocoder extension
-Postgres extension for the CartoDB geocoder. It is meant to contain the functions and related objects needed to provide a geocoding service for administrative areas of level 0, administrative areas of level 1, postal codes, IP addresses and city names. It is not meant to contain the actual data used to geocode them.
+# CARTO geocoder extension
+Postgres extension for the CARTO geocoder. It is meant to contain the functions and related objects needed to provide a geocoding service for administrative areas of level 0, administrative areas of level 1, postal codes, IP addresses and city names. It is not meant to contain the actual data used to geocode them.
 
 ## Dependencies
-This extension is thought to be used on top of CartoDB platform. Therefore **a cartodb user is required** to install the extension onto it.
+This extension is thought to be used on top of CARTO platform. Therefore **a CARTO user is required** to install the extension onto it.
 
 The following is a non-comprehensive list of dependencies:
 
 - Postgres 9.3+
 - Postgis extension
 - Schema triggers extension
-- CartoDB extension
+- cartodb-postgresql CARTO extension
 
-## Installation into the db cluster
-This requires root privileges
+## Installation into the DB cluster
+This step requires root privileges
 ```
 sudo make all install
 ```
 
-## Execute tests
+## Execute the tests
 ```
 PGUSER=postgres make installcheck
 ```
@@ -28,9 +28,9 @@ One-liner:
 sudo PGUSER=postgres make all install installcheck
 ```
 
-## Install onto a cartodb user's database
+## Install onto a CARTO user's database
 
-Remember that **is mandatory to install into a cartodb user's database**
+Remember that **it is mandatory to install it into a CARTO user's database**
 
 ```
 psql -U development_cartodb_user_fe3b850a-01c0-48f9-8a26-a82f09e9b53f cartodb_dev_user_fe3b850a-01c0-48f9-8a26-a82f09e9b53f_db
@@ -42,4 +42,4 @@ and then:
 CREATE EXTENSION cdb_geocoder;
 ```
 
-The extension creation in the user's db does not require special privileges. It can be even created from the sql api.
+The extension creation in the user's DB does not require special privileges. It can be even created from the SQL API.

geocoder/extension/cdb_geocoder--0.0.1--0.0.2.sql

@@ -0,0 +1,58 @@
+CREATE OR REPLACE FUNCTION geocode_admin0_polygons(name text[])
+  RETURNS SETOF geocode_admin_v1 AS $$
+  DECLARE
+    ret geocode_admin_v1%rowtype;
+  BEGIN
+  -- FOR ret IN
+  RETURN QUERY
+    SELECT q, n.the_geom as geom, CASE WHEN s.adm0_a3 IS NULL then FALSE ELSE TRUE END AS success
+    FROM unnest(name) WITH ORDINALITY q
+    LEFT OUTER JOIN admin0_synonyms s ON name_ = lower(geocode_clean_name(q))::text
+    LEFT OUTER JOIN ne_admin0_v3 n ON s.adm0_a3 = n.adm0_a3
+    GROUP BY q, n.the_geom, s.adm0_a3, q.ordinality
+    ORDER BY q.ordinality;
+END
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+CREATE OR REPLACE FUNCTION admin0_synonym_lookup(name text[])
+ RETURNS SETOF synonym_lookup_v1 AS $$
+ DECLARE
+    ret synonym_lookup_v1%rowtype;
+ BEGIN  RETURN QUERY
+    SELECT q, s.adm0_a3
+    FROM unnest(name) WITH ORDINALITY q
+    LEFT OUTER JOIN admin0_synonyms s ON name_ = lower(geocode_clean_name(q))::text
+    GROUP BY q, s.adm0_a3, q.ordinality
+    ORDER BY q.ordinality;
+END
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+CREATE OR REPLACE FUNCTION geocode_namedplace(places text[], admin1s text[], inputcountry text[]) RETURNS SETOF geocode_admin1_country_v1
+    LANGUAGE plpgsql IMMUTABLE SECURITY DEFINER
+    AS $$
+  DECLARE
+    ret geocode_admin1_country_v1%rowtype;
+  BEGIN
+    IF admin1s IS NULL THEN
+      FOR ret IN SELECT g.q as q, NULL as a1, g.c as c, g.geom as geom, g.success as success FROM (SELECT (geocode_namedplace(places, inputcountry)).*) g LOOP
+          RETURN NEXT ret;
+      END LOOP;
+    ELSE
+      FOR ret IN WITH clean AS (
+          SELECT array_agg(p) p, array_agg(a) a, c
+          FROM (SELECT p, a, c
+                FROM (SELECT p, a, c, nest.ordinality as ord FROM unnest(places, admin1s) with ordinality nest (p, a), LATERAL unnest(inputcountry) with ordinality c) z
+                GROUP BY p, a, c, z.ord
+                ORDER BY z.ord
+              ) y
+          GROUP BY c
+        )
+        SELECT (geocode_namedplace(p, a, c)).* FROM clean
+      LOOP
+        RETURN NEXT ret;
+      END LOOP;
+    END IF;
+  RETURN;
+END
+$$;
+

geocoder/extension/cdb_geocoder--0.0.2--0.0.1.sql

@@ -0,0 +1,46 @@
+CREATE OR REPLACE FUNCTION geocode_admin0_polygons(name text[])
+  RETURNS SETOF geocode_admin_v1 AS $$
+  DECLARE
+    ret geocode_admin_v1%rowtype;
+  BEGIN
+  -- FOR ret IN
+  RETURN QUERY
+    SELECT d.q, n.the_geom as geom, CASE WHEN s.adm0_a3 IS NULL then FALSE ELSE TRUE END AS success
+      FROM (SELECT q, lower(geocode_clean_name(q))::text x
+        FROM (SELECT unnest(name) q) g) d
+      LEFT OUTER JOIN admin0_synonyms s ON name_ = d.x
+      LEFT OUTER JOIN ne_admin0_v3 n ON s.adm0_a3 = n.adm0_a3 GROUP BY d.q, n.the_geom, s.adm0_a3;
+END
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+CREATE OR REPLACE FUNCTION admin0_synonym_lookup(name text[])
+ RETURNS SETOF synonym_lookup_v1 AS $$
+ DECLARE
+    ret synonym_lookup_v1%rowtype;
+ BEGIN  RETURN QUERY
+    SELECT d.q, s.adm0_a3
+      FROM (SELECT q, lower(geocode_clean_name(q))::text x
+        FROM (SELECT unnest(name) q) g) d
+      LEFT OUTER JOIN admin0_synonyms s ON name_ = d.x GROUP BY d.q, s.adm0_a3;
+END
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+CREATE OR REPLACE FUNCTION geocode_namedplace(places text[], admin1s text[], inputcountry text[]) RETURNS SETOF geocode_admin1_country_v1
+    LANGUAGE plpgsql IMMUTABLE SECURITY DEFINER
+    AS $$
+  DECLARE
+    ret geocode_admin1_country_v1%rowtype;
+  BEGIN
+    IF admin1s IS NULL THEN
+      FOR ret IN SELECT g.q as q, NULL as a1, g.c as c, g.geom as geom, g.success as success FROM (SELECT (geocode_namedplace(places, inputcountry)).*) g LOOP
+          RETURN NEXT ret;
+      END LOOP;
+    ELSE
+      FOR ret IN WITH clean AS (SELECT array_agg(p) p, array_agg(a) a, c FROM (SELECT p, a, c FROM (SELECT unnest(places) p, unnest(admin1s) a, unnest(inputcountry) c) z GROUP BY p, a, c) y GROUP BY c)
+        SELECT (geocode_namedplace(p, a, c)).* FROM clean LOOP
+          RETURN NEXT ret;
+      END LOOP;
+    END IF;
+  RETURN;
+END
+$$;

geocoder/extension/cdb_geocoder.control

@@ -1,6 +1,6 @@
 # cdb geocoder extension
 comment = 'CartoDB internal geocoder'
-default_version = '0.0.1'
+default_version = '0.0.2'
 relocatable = true
 requires = cartodb
 superuser = false

geocoder/extension/sql/20_admin0.sql

@@ -8,11 +8,12 @@ CREATE OR REPLACE FUNCTION geocode_admin0_polygons(name text[])
   BEGIN
   -- FOR ret IN
   RETURN QUERY
-    SELECT d.q, n.the_geom as geom, CASE WHEN s.adm0_a3 IS NULL then FALSE ELSE TRUE END AS success
-      FROM (SELECT q, lower(geocode_clean_name(q))::text x
-        FROM (SELECT unnest(name) q) g) d
-      LEFT OUTER JOIN admin0_synonyms s ON name_ = d.x
-      LEFT OUTER JOIN ne_admin0_v3 n ON s.adm0_a3 = n.adm0_a3 GROUP BY d.q, n.the_geom, s.adm0_a3;
+    SELECT q, n.the_geom as geom, CASE WHEN s.adm0_a3 IS NULL then FALSE ELSE TRUE END AS success
+    FROM unnest(name) WITH ORDINALITY q
+    LEFT OUTER JOIN admin0_synonyms s ON name_ = lower(geocode_clean_name(q))::text
+    LEFT OUTER JOIN ne_admin0_v3 n ON s.adm0_a3 = n.adm0_a3
+    GROUP BY q, n.the_geom, s.adm0_a3, q.ordinality
+    ORDER BY q.ordinality;
 END
 $$ LANGUAGE 'plpgsql' SECURITY DEFINER;
 
@@ -23,10 +24,11 @@ CREATE OR REPLACE FUNCTION admin0_synonym_lookup(name text[])
  DECLARE
     ret synonym_lookup_v1%rowtype;
  BEGIN  RETURN QUERY
-    SELECT d.q, s.adm0_a3
-      FROM (SELECT q, lower(geocode_clean_name(q))::text x
-        FROM (SELECT unnest(name) q) g) d
-      LEFT OUTER JOIN admin0_synonyms s ON name_ = d.x GROUP BY d.q, s.adm0_a3;
+    SELECT q, s.adm0_a3
+    FROM unnest(name) WITH ORDINALITY q
+    LEFT OUTER JOIN admin0_synonyms s ON name_ = lower(geocode_clean_name(q))::text
+    GROUP BY q, s.adm0_a3, q.ordinality
+    ORDER BY q.ordinality;
 END
 $$ LANGUAGE 'plpgsql' SECURITY DEFINER;
 

geocoder/extension/sql/50_namedplaces.sql

@@ -225,7 +225,7 @@ CREATE OR REPLACE FUNCTION geocode_namedplace(places text[], admin1s text[], inp
       next AS (SELECT p.s AS q, p.a1 AS a1, (SELECT gp.the_geom FROM global_cities_points_limited gp, global_cities_alternates_limited ga WHERE lower(p.s) = ga.lowername AND ga.admin1 = p.i AND ga.geoname_id = gp.geoname_id ORDER BY preferred DESC LIMIT 1) geom FROM p WHERE p.s NOT IN (SELECT q FROM best WHERE geom IS NOT NULL))
       SELECT q, a1, inputcountry as c, geom, TRUE AS success FROM best WHERE geom IS NOT NULL
       UNION ALL
-      SELECT q, a1, inputcountry as c, geom, CASE WHEN geom IS NULL THEN FALSE ELSE TRUE END AS success FROM next 
+      SELECT q, a1, inputcountry as c, geom, CASE WHEN geom IS NULL THEN FALSE ELSE TRUE END AS success FROM next
       LOOP
       RETURN NEXT ret;
     END LOOP;
@@ -260,9 +260,18 @@ CREATE OR REPLACE FUNCTION geocode_namedplace(places text[], admin1s text[], inp
           RETURN NEXT ret;
       END LOOP;
     ELSE
-      FOR ret IN WITH clean AS (SELECT array_agg(p) p, array_agg(a) a, c FROM (SELECT p, a, c FROM (SELECT unnest(places) p, unnest(admin1s) a, unnest(inputcountry) c) z GROUP BY p, a, c) y GROUP BY c)
-        SELECT (geocode_namedplace(p, a, c)).* FROM clean LOOP
-          RETURN NEXT ret;
+      FOR ret IN WITH clean AS (
+          SELECT array_agg(p) p, array_agg(a) a, c
+          FROM (SELECT p, a, c
+                FROM (SELECT p, a, c, nest.ordinality as ord FROM unnest(places, admin1s) with ordinality nest (p, a), LATERAL unnest(inputcountry) with ordinality c) z
+                GROUP BY p, a, c, z.ord
+                ORDER BY z.ord
+              ) y
+          GROUP BY c
+        )
+        SELECT (geocode_namedplace(p, a, c)).* FROM clean
+      LOOP
+        RETURN NEXT ret;
       END LOOP;
     END IF;
   RETURN;

geocoder/extension/test/expected/00_install_test.out

@@ -1,8 +1,14 @@
 -- Install dependencies
 CREATE EXTENSION postgis;
-CREATE EXTENSION schema_triggers;
 CREATE EXTENSION plpythonu;
 CREATE EXTENSION cartodb;
+-- Mock the quota check
+SELECT cartodb.CDB_SetUserQuotaInBytes(0);
+ cdb_setuserquotainbytes 
+-------------------------
+                       0
+(1 row)
+
 -- Install the extension
 CREATE EXTENSION cdb_geocoder;
 -- Mock the varnish invalidation function

geocoder/extension/test/expected/20_admin0_test.out

@@ -10,9 +10,9 @@ SELECT (admin0_synonym_lookup(Array['United States', 'ESP'])).*;
 SELECT (geocode_admin0_polygons(Array['Spain', 'USA', ''])).*;
    q   | geom | success 
 -------+------+---------
-       |      | f
  Spain |      | f
  USA   |      | f
+       |      | f
 (3 rows)
 
 -- Add a few synonyms
@@ -21,7 +21,7 @@ COPY admin0_synonyms (name, rank, created_at, updated_at, the_geom, the_geom_web
 SELECT (admin0_synonym_lookup(Array['United States', 'ESP'])).*;
        q       | adm0_a3 
 ---------------+---------
- ESP           | ESP
  United States | USA
+ ESP           | ESP
 (2 rows)
 

geocoder/extension/test/expected/50_namedplaces_test.out

@@ -42,8 +42,8 @@ SELECT (geocode_namedplace(Array['Portland', 'Portland', 'New York City'], Array
        q       |   a1   |  c  | geom | success 
 ---------------+--------+-----+------+---------
  New York City |        | USA |      | f
- Portland      | Oregon | USA |      | f
  Portland      | Maine  | USA |      | f
+ Portland      | Oregon | USA |      | f
 (3 rows)
 
 SELECT namedplace_guess_country(Array['granada', 'jaen', 'cordoba', 'madrid', 'valladolid']);

geocoder/extension/test/sql/00_install_test.sql

@@ -1,9 +1,11 @@
 -- Install dependencies
 CREATE EXTENSION postgis;
-CREATE EXTENSION schema_triggers;
 CREATE EXTENSION plpythonu;
 CREATE EXTENSION cartodb;
 
+-- Mock the quota check
+SELECT cartodb.CDB_SetUserQuotaInBytes(0);
+
 -- Install the extension
 CREATE EXTENSION cdb_geocoder;
 

geocoder/geocoder_apply_patches.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+function usage() {
+  cat <<EOF
+Usage:
+  $(basename $0) DBUSER DBNAME *.sql
+E.g:
+  $(basename $0) development_cartodb_user_87ddf981-25c7-4538-9910-0eb4342f2483 cartodb_dev_user_87ddf981-25c7-4538-9910-0eb4342f2483_db patches/*.sql
+EOF
+}
+
+
+if [ "$#" -lt "3" ]; then
+    usage
+    exit 1
+fi
+
+DBUSER=$1
+DBNAME=$2
+shift; shift;
+DUMP_FILES="$@"
+
+echo
+echo "About to import the following files: ${DUMP_FILES}"
+for i in $DUMP_FILES; do
+    echo
+    echo "Importing ${i}..."
+    psql \
+        --username=${DBUSER} \
+        --dbname=${DBNAME} \
+        --set=ON_ERROR_STOP=on \
+        --single-transaction \
+        --file=${i} || exit 1
+    echo "Done with ${i}."
+    echo
+done
+
+echo
+echo "** Everything OK **"

geocoder/geocoder_download_dumps

@@ -21,5 +21,5 @@ mkdir -p $TARGET_DIR
 for file in $DUMP_LIST; do
     url="${BASE_URL}/${VERSION}/$file"
 
-    wget --directory-prefix=$TARGET_DIR $url
+    wget -c --directory-prefix=$TARGET_DIR $url
 done

geocoder/geocoder_download_patches.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+
+TARGET_DIR_PATCHES=data_patches
+BASE_URL=https://s3.amazonaws.com/data.cartodb.net/geocoding/dumps
+VERSION=0.0.1
+
+PATCHES_LIST="20160203_countries_bh_isocode.sql
+20160622_countries_synonym_congo.sql
+20171004_merge_corsica_and_france.sql
+20180117_hsinchu_synonyms.sql
+20180306_add_ssd_rows_for_south_sudan.sql
+20181011_add_synonyms_for_swaziland.sql
+20190111_france_regions_typos.sql
+20210118_add_renamed_country_north_macedonia.sql
+20220325_france_region_haut-rhin_typo.sql"
+
+mkdir -p $TARGET_DIR_PATCHES
+
+for file in $PATCHES_LIST; do
+    url="${BASE_URL}/${VERSION}/patches/$file"
+
+    wget -c --directory-prefix=$TARGET_DIR_PATCHES $url
+done

geocoder/ip-addresses/README.md

@@ -11,12 +11,61 @@ SELECT geocode_ip(Array['1.0.16.0', '::ffff:1.0.16.0'])
 ```
 
 # Creation steps
-1. Create the `ip_address_locations` table
+1. Create the `ip_address_locations` table (see `40_ipaddr.sql` file)
 2. Obtain the file from http://geolite.maxmind.com/download/geoip/database/GeoLite2-City-CSV.zip
 3. Uncompress it and upload the `GeoLite2-City-Blocks-IPv4.csv` file
 4. Rename the uploaded table as `latest_ip_address_locations`
 5. Run the `sql/build_data_table` script to update the table
 
+# Update steps
+
+## Option A: generate a new `ip_address_locations` table at geocoder user
+
+If the geocoder database is a CARTO user, do these steps:
+
+1. Import `GeoLite2-City-Blocks-IPv4.csv` and rename it to `latest_ip_address_locations`.
+2. Import `GeoLite2-City-Blocks-IPv6.csv` and rename it to `latest_ip6_address_locations`.
+3. If you want to create a backup of the previous table, do this:
+
+    ```sql
+    CREATE TABLE ip_address_locations_backup as
+        select * from ip_address_locations;
+    ```
+
+4. Clear previous table:
+
+    ```sql
+    TRUNCATE ip_address_locations;
+    ```
+
+5. Load the new values:
+
+```sql
+set statement_timeout = '20min';
+INSERT INTO ip_address_locations (the_geom, network_start_ip) SELECT the_geom, ('::ffff:' || split_part(network, '/', 1))::inet FROM latest_ip_address_locations;
+INSERT INTO ip_address_locations (the_geom, network_start_ip) SELECT the_geom, split_part(network, '/', 1)::inet FROM latest_ip6_address_locations;
+```
+
+## Option B: load a dump of the table
+
+If the geocoder database is not a CARTO user, do these steps:
+
+1. Perform option A in any user (it can even be a staging user). If you need to create the table, check `40_ipaddr.sql`.
+2. Generate a dump of the file at that user database:
+
+    ```sql
+    \copy (select * from ip_address_locations) TO /tmp/ip_address_locations.dump;
+    ```
+
+3. Copy the file to the remote server.
+4. Perform steps A.3 and A.4.
+5. Load the new data (takes ~10 minutes):
+
+```sql
+set statement_timeout = '20min';
+\copy ip_address_locations from /tmp/ip_address_locations.dump
+```
+
 # Tables
 
 ### ip_address_locations
@@ -65,6 +114,8 @@ In order to test the data and the functions created under the script avaialble i
 # Known issues
 
 # Historic:
+* [01/12/2015]:
+  * Removed geocoder function. Check /extensions instead.
 * [19/10/2015]:
   * Updates README and adds usage example and definition of the service
 * [08/10/2015]:

geocoder/ip-addresses/sql/build_data_table.sql

@@ -4,5 +4,5 @@
 -- Clear table
 DELETE FROM ip_address_locations;
 -- Updates table with new source data
-INSERT INTO ip_address_locations (the_geom, network_start_ip) SELECT the_geom, network_start_ip::inet FROM latest_ip_address_locations;
+INSERT INTO ip_address_locations (the_geom, network_start_ip) SELECT the_geom, ('::ffff:' || split_part(network, '/', 1))::inet FROM latest_ip_address_locations;
 DROP TABLE latest_ip_address_locations;

geocoder/namedplace/README.md

@@ -13,7 +13,7 @@ SELECT (geocode_namedplace(Array['sunapee', 'sunapeeee', 'New York City'], 'USA'
 
 # Creation steps
 1. Download the `allCountries` and `alternateNames` tables from the source
-2. In order to clean the files from characters that may provoke issues in the importation to CartoDB, please run in your terminal: `sed 's/"//' filename.txt > filename.out.txt`
+2. In order to clean the files from characters that may provoke issues when importing to CARTO, please run in your terminal: `sed 's/"//' filename.txt > filename.out.txt`
 3. Import the zipped files for `allCountries` and `alternateNames` once processed
 4. Generate the `global_cities_points_limited` and `global_cities_alternates_limited` tables
 5. Run the `sql/build_data_table.sql` script to build the `global_cities_points_limited` table
@@ -165,6 +165,8 @@ In order to test the data and the functions created under the script avaialble i
 * Needs a better approach for synonyms and normalisation
 
 # Historic:
+* [01/12/2015]:
+  * Removed geocoder function. Check /extensions instead.
 * [19/10/2015]:
   * Updates README with process description and update setup scripts
 * [08/10/2015]:

geocoder/postal-codes/README.md

@@ -16,7 +16,7 @@ SELECT (geocode_postalcode_polygons(Array['11211'],Array['USA'])).*
 
 ## Creation steps
 
-1. Import the four files attached in the section "Datasources" for Australia (`doc` table), Canada (`gfsa000a11a_e` table), USA (`tl_2013_us_zcta510` table) and France (`codes_postaux` table).
+1. Import the four files attached in the section "Datasources" for Australia (`doc` table), Canada (`gfsa000a11a_e` table), USA (`tl_2013_us_zcta510` table) and France (`codes_postaux` table, renamed from `codes_postaux_region`).
 
 2. Run `sql/build_data_table.sql`. Notice that table `postal_code_polygons` should exist in advance with columns: `the_geom`, `adm0_a3` and `postal_code`.
 
@@ -187,8 +187,10 @@ SELECT (geocode_postalcode_points(Array['03204'],Array['ESP'])).*
   ![Duplicates](https://camo.githubusercontent.com/1dbd4874830b0654b2fc2e11cd2a650d498f6bc9/68747470733a2f2f646c2e64726f70626f7875736572636f6e74656e742e636f6d2f752f323837393330382f53637265656e25323053686f74253230323031352d30362d3239253230617425323031322e35362e30332e706e67)
 
 # Historic:
+* [01/12/2015]:
+  * Removed geocoder function. Check /extensions instead.
 * [30/10/2015]:
-  * Found bug #173 https://github.com/CartoDB/data-services/issues/173 
+  * Found bug #173 https://github.com/CartoDB/data-services/issues/173
 * [19/10/2015]:
   * Updates readme with usage examples and setup scripts
 * [08/10/2015]: