Release 1.0.2 artifact

Release 1.0.2 wit mocks for augment functions

NEWS.md

@@ -1,3 +1,89 @@
-0.0.1 (open date)
+1.0.2 (2016-07-12)
+---
+
+__Bugfixes__
+
+* Fix for `OBS_GetCategory` outside the US ([#135](https://github.com/CartoDB/observatory-extension/pull/137))
+* `OBS_GetMeasure` now respects the `normalize` parameter even when passed
+  a multi/polygon. Previously, no normalization was erroneously assumed.
+
+__Improvements__
+
+* Automated tests cover Mexico data
+* `obs_meta` is now provisioned during unit tests
+* `obs_meta` is now used during end-to-end tests
+* `OBS_GetMeasureByID` uses `obs_meta` internally, which should help
+  performance
+* `OBS_GetCategory` uses `obs_meta` internally, which should help perfromance
+* `OBS_GetCategory` will pick the correct category for an arbitrary polygon
+  (the category covering the highest % of that polygon)
+* `OBS_GetMeasure` has been updated to use `obs_meta` internally, which should
+  help performance
+* `OBS_GetMeasure` now can be passed "none" and skip normalization by area or
+  denominator for points
+* Fixtures are only loaded at the start of the unit test suite, and dropped at the end,
+  instead of at the start/end of each individual test file
+* Comment noisy NOTICEs ([#73](https://github.com/CartoDB/observatory-extension/issues/73))
+
+1.0.1 (2016-07-01)
+---
+
+__Bugfixes__
+
+* Fix for ERROR:  Operation on mixed SRID geometries #130
+
+
+1.0.0 (6/27/2016)
+-----
+
+* Incremented to 1.0.0 to be in compliance with [SemVer](http://semver.org/),
+  which disallows use of 0.x.x versions.  This also reflects that we are
+  already in production.
+
+__API Changes__
+
+* Added `OBS_DumpVersion` to look up version data ([#118](https://github.com/CartoDB/observatory-extension/pull/118))
+
+__Improvements__
+
+* Whether data exists for a geom now determined by polygon intersection instead of
+  BBOX overlap ([#119](https://github.com/CartoDB/observatory-extension/pull/119))
+* Automated tests cover Spanish and UK data
+  ([#115](https://github.com/CartoDB/observatory-extension/pull/115))
+* Automated tests cover `OBS_GetUSCensusMeasure`
+  ([#105](https://github.com/CartoDB/observatory-extension/pull/105))
+
+__Bugfixes__
+
+* Geom table can have different `geomref_colname` than the data table
+  ([#123](https://github.com/CartoDB/observatory-extension/pull/123))
+
+
+0.0.5 (5/27/2016)
+-----
+* Adds new function `OBS_GetMeasureById` ([#96](https://github.com/CartoDB/observatory-extension/pull/96))
+
+0.0.4 (5/25/2016)
+-----
+* Updates queries involving US Census measure tags to be more generic ([#95](https://github.com/CartoDB/observatory-extension/pull/95))
+* Fixes tests which relied on an erroneous subset of block groups ([#95](https://github.com/CartoDB/observatory-extension/pull/95))
+
+0.0.3 (5/24/2016)
+-----
+* Generalizes internal queries to properly pull from multiple named geometry references
+* Adds tests for Who's on First boundaries
+* Improves automatic fixtures testing script
+
+0.0.2 (5/19/2016)
+-----
+* Adds Data Observatory exploration functions
+* Adds Data Observatory boundary functions
+* Adds Data Observatory measure functions
+* Adds script to generate fixtures for tests
+* Adds script for the automatic testing of metadata
+* Adds full documentation for all included functions
+* removes `cartodb` extension dependency
+
+0.0.1 (5/19/2016)
 ------------------
-* First iteration of `OBS_GetDemographicSnapshot(location Geometry(Point,4326))`;
+* First iteration of `OBS_GetDemographicSnapshot(location Geometry(Point,4326))`

RELEASE.md

@@ -20,12 +20,6 @@ script for the new release, `release/observatory--X.Y.Z.sql`:
 make release
 ```
 
-Then, the release manager shall produce upgrade and downgrade scripts
-to migrate to/from the previous release. In the case of minor/patch
-releases this simply consist in extracting the functions that have changed
-and placing them in the proper `release/observatory--X.Y.Z--A.B.C.sql`
-file.
-
 The new release can be deployed for staging/smoke tests with this command:
 
 ```

doc/API.md

@@ -4,6 +4,8 @@ This file is for reference purposes only. It is intended for tracking the Data O
 
 ## Documentation
 
+* Overview (local file in the Docs repo)
+* Accessing the Data Observatory (local file in the Docs repo)
 * [Measures Functions](measures_functions.md)
 * [Boundary Functions](boundary_functions.md)
 * [Discovery Functions](discovery_functions.md)

doc/boundary_functions.md

@@ -1,8 +1,8 @@
 # Boundary Functions
 
-Use the following functions to retrieve [Boundary](/cartodb-platform/data/overview/#boundary-data) data. Data ranges from small areas (e.g. US Census Block Groups) to large areas (e.g. Countries). You can access boundaries by point location lookup, bounding box lookup, direct ID access and several other methods described below.
+Use the following functions to retrieve [Boundary](/carto-engine/data/overview/#boundary-data) data. Data ranges from small areas (e.g. US Census Block Groups) to large areas (e.g. Countries). You can access boundaries by point location lookup, bounding box lookup, direct ID access and several other methods described below.
 
-You can [access](/cartodb-platform/data/accessing/#accessing-the-data-observatory) boundaries through the CartoDB Editor. The same methods will work if you are using the CartoDB Platform to develop your application. We [encourage you](/cartodb-platform/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
+You can [access](/carto-engine/data/accessing/#accessing-the-data-observatory) boundaries through the CARTO Editor. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
 
 ## OBS_GetBoundariesByGeometry(polygon geometry, geometry_id text)
 

doc/discovery_functions.md

@@ -1,6 +1,6 @@
 # Discovery Functions
 
-If you are using the [discovery methods](/cartodb-platform/data/overview/#discovery-methods) from the Data Observatory, use the following functions to retrieve [boundary](/cartodb-platform/data/overview/#boundary-data) and [measures](/cartodb-platform/data/overview/#measures-data) data.
+If you are using the [discovery methods](/carto-engine/data/overview/#discovery-methods) from the Data Observatory, use the following functions to retrieve [boundary](/carto-engine/data/overview/#boundary-data) and [measures](/carto-engine/data/overview/#measures-data) data.
 
 ## OBS_Search(search_term)
 
@@ -47,7 +47,7 @@ A TABLE containing the following properties
 
 Key | Description
 --- | ---
-boundary_id | a boundary identifier from the [boundary ID glossary](/cartodb-platform/data/glossary/#boundary-ids)
+boundary_id | a boundary identifier from the [boundary ID glossary](/carto-engine/data/glossary/#boundary-ids)
 description | a brief description of the boundary dataset
 time_span | the timespan attached the boundary. this does not mean that the boundary is invalid outside of the timespan, but is the explicit timespan published with the geometry.
 

doc/glossary.md

@@ -29,7 +29,7 @@ Disputed Areas  |  whosonfirst.wof_disputed_geom  |  none
 
 ## OBS_GetUSCensusMeasure Names Table
 
-This list contains human readable names accepted in the ```OBS_GetUSCensusMeasure``` function. For the more comprehensive list of columns available to the ```OBS_GetMeasure``` function, see the [Data Observatory Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf).
+This list contains human readable names accepted in the ```OBS_GetUSCensusMeasure``` function. For the more comprehensive list of columns available to the ```OBS_GetMeasure``` function, see the [Data Observatory Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf).
 
 Measure name             |  Measure description
 ------------------------ | --------------------

doc/license.md

@@ -1,6 +1,8 @@
 # License
 
-The Data Observatory is a collection of various sources of data with varying licenses. We have worked hard to find you data that will work for the broadest set of use-cases. For competency, please still review the terms for any dataset you use and respect the rights of the owners for each dataset. The following third-party data sources are used in the Data Observatory, and we have included the links to the terms governing their use.
+The Data Observatory is a collection of data sources with varying licenses and terms of use. We have endeavored to find you data that will work for the broadest set of use-cases. The following third-party data sources are used in the Data Observatory, and we have included the links to the terms governing their use.
+
+_**Legal Note**: The Data Observatory makes use of a variety of third party data and databases (collectively, the “Data”). You acknowledge that the included Data, and the licenses and terms of use, may be amended from time to time. Whenever you use the Data, you agree to the current relevant terms or license. Some Data will require that you provide attribution to the data source. Other Data may be protected by US or international copyright laws, treaties, or conventions. The Data and associated metadata are provided 'as-is', without express or implied warranty of any kind, including, but not limited to, infringement, merchantability and fitness for a particular purpose. CartoDB is not responsible for the accuracy, completeness, timeliness or quality of the Data._
 
 Name  | Terms link 
 -------|---------
@@ -14,5 +16,5 @@ Natural Earth  |  [http://www.naturalearthdata.com/about/terms-of-use/](http://w
 Quattroshapes  |  [https://github.com/foursquare/quattroshapes/blob/master/LICENSE.md](https://github.com/foursquare/quattroshapes/blob/master/LICENSE.md)
 Zetashapes  |  [http://zetashapes.com/license](http://zetashapes.com/license)
 Spielman & Singleton  |  [https://www.openicpsr.org/repoEntity/show/41329](https://www.openicpsr.org/repoEntity/show/41329)
-Instituto Nacional de Estadistica  |  [http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout](http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout)
+El Instituto Nacional de Estadística (INE) | The National Statistics Institute (INE) of Spain includes data from multiple sources. If you are re-using their data, they explicitly require that you reference them accordingly<br /><br />[http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout](http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout)
 

doc/measures_functions.md

@@ -1,10 +1,10 @@
 # Measures Functions
 
-[Data Observatory Measures](/cartodb-platform/data/overview/#measures-methods) are the numerical location data you can access. The measure functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (point or polygon) and receiving back a measure (an absolute value) for that location.
+[Data Observatory Measures](/carto-engine/data/overview/#measures-methods) are the numerical location data you can access. The measure functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (point or polygon) and receiving back a measure (an absolute value) for that location.
 
-There are hundreds of measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf).
+There are hundreds of measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf).
 
-You can [access](/cartodb-platform/data/accessing/#accessing-the-data-observatory) measures through the CartoDB Editor. The same methods will work if you are using the CartoDB Platform to develop your application. We [encourage you](/cartodb-platform/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
+You can [access](/carto-engine/data/accessing/#accessing-the-data-observatory) measures through the CARTO Editor. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
 
 ## OBS_GetUSCensusMeasure(point geometry, measure_name text)
 
@@ -16,7 +16,7 @@ Name |Description
 --- | ---
 point | a WGS84 point geometry (the_geom)
 measure_name | a human readable name of a US Census variable. The list of measure_names is [available in the glossary](/cartodb-platform/data/glossary/#obsgetuscensusmeasure-names-table).
-normalize | for measures that are are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://cartodb.github.io/bigmetadata/index.html) (optional)
+normalize | for measures that are are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf) (optional)
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span | time span of interest (e.g., 2010 - 2014)
 
@@ -47,7 +47,7 @@ Name |Description
 --- | ---
 polygon | a WGS84 polygon geometry (the_geom)
 measure_name | a human readable string name of a US Census variable. The list of measure_names is [available in the glossary](/cartodb-platform/data/glossary/#obsgetuscensusmeasure-names-table).
-normalize | for measures that are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf) (optional)
+normalize | for measures that are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf) (optional)
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span | time span of interest (e.g., 2010 - 2014)
 
@@ -70,7 +70,7 @@ SET local_male_population = OBS_GetUSCensusMeasure(the_geom, 'Male Population')
 
 ## OBS_GetMeasure(point geometry, measure_id text)
 
-The ```OBS_GetMeasure(point, measure_id)``` function returns any Data Observatory measure at a point location. You can browse all available Measures in the [Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf)).
+The ```OBS_GetMeasure(point, measure_id)``` function returns any Data Observatory measure at a point location. You can browse all available Measures in the [Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf).
 
 #### Arguments
 
@@ -78,7 +78,7 @@ Name |Description
 --- | ---
 point | a WGS84 point geometry (the_geom)
 measure_id | a measure identifier from the Data Observatory ([see available measures](https://cartodb.github.io/bigmetadata/observatory.pdf)). It is important to note that these are different than 'measure_name' used in the Census based functions above.
-normalize | for measures that are are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. The other option is 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf). (optional)
+normalize | for measures that are are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. The other option is 'denominator', which will use the denominator specified in the [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf). (optional)
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span | time span of interest (e.g., 2010 - 2014)
 
@@ -109,7 +109,7 @@ Name |Description
 --- | ---
 polygon_geometry | a WGS84 polygon geometry (the_geom)
 measure_id | a measure identifier from the Data Observatory ([see available measures](https://cartodb.github.io/bigmetadata/observatory.pdf))  
-normalize | for measures that are are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf) (optional)
+normalize | for measures that are are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf) (optional)
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span | time span of interest (e.g., 2010 - 2014)
 
@@ -134,9 +134,43 @@ SET household_count = OBS_GetMeasure(the_geom, 'us.census.acs.B11001001')
 
 * If an unrecognized normalization type is input, raise an error: `'Only valid inputs for "normalize" are "area" (default) and "denominator".`
 
+## OBS_GetMeasureById(geom_ref text, measure_id text, boundary_id text)
+
+The ```OBS_GetMeasureById(geom_ref, measure_id, boundary_id)``` function returns any Data Observatory measure that corresponds to the boundary in ```boundary_id``` that has a geometry reference of ```geom_ref```.
+
+#### Arguments
+
+Name |Description
+--- | ---
+geom_ref | a geometry reference (e.g., a US Census geoid)
+measure_id | a measure identifier from the Data Observatory ([see available measures](https://cartodb.github.io/bigmetadata/observatory.pdf))  
+boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
+time_span (optional) | time span of interest (e.g., 2010 - 2014). If `NULL` is passed, the measure from the most recent data will be used.
+
+#### Returns
+
+A NUMERIC value
+
+Key | Description
+--- | ---
+value | the raw measure associated with `geom_ref`
+
+#### Example
+
+Add a measure to an empty column based on county geoids in your table
+
+```SQL
+UPDATE tablename
+SET household_count = OBS_GetMeasureById(geoid_column, 'us.census.acs.B11001001', 'us.census.tiger.county')
+```
+
+#### Errors
+
+* Returns `NULL` if there is a mismatch between the geometry reference and the boundary id such as using the geoid of a county with the boundary of block groups
+
 ## OBS_GetCategory(point geometry, category_id text)
 
-The ```OBS_GetCategory(point, category_id)``` function returns any Data Observatory Category value at a point location. The Categories available are currently limited to Segmentation categories. See the Segmentation section of the [Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf) for more detail.
+The ```OBS_GetCategory(point, category_id)``` function returns any Data Observatory Category value at a point location. The Categories available are currently limited to Segmentation categories. See the Segmentation section of the [Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf) for more detail.
 
 #### Arguments
 

release/observatory.control

@@ -1,5 +1,5 @@
 comment = 'CartoDB Observatory backend extension'
-default_version = '0.0.2'
+default_version = '1.0.2'
 requires = 'postgis'
 superuser = true
 schema = cdb_observatory

scripts/autotest.py

@@ -24,16 +24,18 @@ def query(q, is_meta=False, **options):
     params['api_key'] = META_API_KEY if is_meta else API_KEY
     return requests.get(url, params=params)
 
-MEASURE_COLUMNS = [(r['id'], ) for r in query('''
-SELECT id FROM obs_column
-WHERE type ILIKE 'numeric'
-AND weight > 0
+MEASURE_COLUMNS = [(r['numer_id'], r['point_only'], ) for r in query('''
+SELECT distinct numer_id, numer_aggregate NOT ILIKE 'sum' as point_only
+FROM obs_meta
+WHERE numer_type ILIKE 'numeric'
+AND numer_weight > 0
 ''', is_meta=True).json()['rows']]
 
-CATEGORY_COLUMNS = [(r['id'], ) for r in query('''
-SELECT id FROM obs_column
-WHERE type ILIKE 'text'
-AND weight > 0
+CATEGORY_COLUMNS = [(r['numer_id'], ) for r in query('''
+SELECT distinct numer_id
+FROM obs_meta
+WHERE numer_type ILIKE 'text'
+AND numer_weight > 0
 ''', is_meta=True).json()['rows']]
 
 BOUNDARY_COLUMNS = [(r['id'], ) for r in query('''
@@ -42,6 +44,36 @@ WHERE type ILIKE 'geometry'
 AND weight > 0
 ''', is_meta=True).json()['rows']]
 
+US_CENSUS_MEASURE_COLUMNS = [(r['numer_name'], ) for r in query('''
+SELECT distinct numer_name
+FROM obs_meta
+WHERE numer_type ILIKE 'numeric'
+AND 'us.census.acs.acs' = ANY (subsection_tags)
+AND numer_weight > 0
+''', is_meta=True).json()['rows']]
+
+
+def default_geometry_id(column_id):
+    '''
+    Returns default test point for the column_id.
+    '''
+    if column_id == 'whosonfirst.wof_disputed_geom':
+        return 'CDB_LatLng(33.78, 76.57)'
+    elif column_id == 'whosonfirst.wof_marinearea_geom':
+        return 'CDB_LatLng(43.33, -68.47)'
+    elif column_id in ('us.census.tiger.school_district_elementary',
+                       'us.census.tiger.school_district_secondary',
+                       'us.census.tiger.school_district_elementary_clipped',
+                       'us.census.tiger.school_district_secondary_clipped'):
+        return 'CDB_LatLng(40.7025, -73.7067)'
+    elif column_id.startswith('es.ine'):
+        return 'CDB_LatLng(42.8226119029222, -2.51141249535454)'
+    elif column_id.startswith('us.zillow'):
+        return 'CDB_LatLng(28.3305906291771, -81.3544048197256)'
+    else:
+        return 'CDB_LatLng(40.7, -73.9)'
+
+
 def default_point(column_id):
     '''
     Returns default test point for the column_id.
@@ -55,16 +87,63 @@ def default_point(column_id):
                        'us.census.tiger.school_district_elementary_clipped',
                        'us.census.tiger.school_district_secondary_clipped'):
         return 'CDB_LatLng(40.7025, -73.7067)'
-    elif column_id.startswith('es.ine'):
-        return 'CDB_LatLng(40.39, -3.7)'
+    elif column_id.startswith('uk'):
+        if 'WA' in column_id:
+            return 'CDB_LatLng(51.46844551219723, -3.184833526611328)'
+        else:
+            return 'CDB_LatLng(51.51461834694225, -0.08883476257324219)'
+    elif column_id.startswith('es'):
+        return 'CDB_LatLng(42.8226119029222, -2.51141249535454)'
     elif column_id.startswith('us.zillow'):
         return 'CDB_LatLng(28.3305906291771, -81.3544048197256)'
+    elif column_id.startswith('mx.'):
+        return 'CDB_LatLng(19.41347699386547, -99.17019367218018)'
     else:
         return 'CDB_LatLng(40.7, -73.9)'
 
 
+def default_area(column_id):
+    '''
+    Returns default test area for the column_id
+    '''
+    point = default_point(column_id)
+    area = 'ST_Transform(ST_Buffer(ST_Transform({point}, 3857), 1000), 4326)'.format(
+        point=point)
+    return area
+
+@parameterized(US_CENSUS_MEASURE_COLUMNS)
+def test_get_us_census_measure_points(name):
+    print 'test_get_us_census_measure_points, ', name
+    resp = query('''
+SELECT * FROM {schema}OBS_GetUSCensusMeasure({point}, '{name}')
+                 '''.format(name=name.replace("'", "''"),
+                            schema='cdb_observatory.' if USE_SCHEMA else '',
+                            point=default_point('')))
+    assert_equal(resp.status_code, 200)
+    rows = resp.json()['rows']
+    assert_equal(1, len(rows))
+    assert_is_not_none(rows[0].values()[0])
+
+
 @parameterized(MEASURE_COLUMNS)
-def test_measure_points(column_id):
+def test_get_measure_areas(column_id, point_only):
+    print 'test_get_measure_areas, ', column_id, point_only
+    if point_only:
+        return
+    resp = query('''
+SELECT * FROM {schema}OBS_GetMeasure({area}, '{column_id}')
+                 '''.format(column_id=column_id,
+                            schema='cdb_observatory.' if USE_SCHEMA else '',
+                            area=default_area(column_id)))
+    assert_equal(resp.status_code, 200)
+    rows = resp.json()['rows']
+    assert_equal(1, len(rows))
+    assert_is_not_none(rows[0].values()[0])
+
+
+@parameterized(MEASURE_COLUMNS)
+def test_get_measure_points(column_id, point_only):
+    print 'test_get_measure_points, ', column_id, point_only
     resp = query('''
 SELECT * FROM {schema}OBS_GetMeasure({point}, '{column_id}')
                  '''.format(column_id=column_id,
@@ -75,8 +154,21 @@ SELECT * FROM {schema}OBS_GetMeasure({point}, '{column_id}')
     assert_equal(1, len(rows))
     assert_is_not_none(rows[0].values()[0])
 
+#@parameterized(CATEGORY_COLUMNS)
+#def test_get_category_areas(column_id):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetCategory({area}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            area=default_area(column_id)))
+#    assert_equal(resp.status_code, 200)
+#    rows = resp.json()['rows']
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0].values()[0])
+
 @parameterized(CATEGORY_COLUMNS)
-def test_category_points(column_id):
+def test_get_category_points(column_id):
+    print 'test_get_category_points, ', column_id
     resp = query('''
 SELECT * FROM {schema}OBS_GetCategory({point}, '{column_id}')
                  '''.format(column_id=column_id,
@@ -87,14 +179,62 @@ SELECT * FROM {schema}OBS_GetCategory({point}, '{column_id}')
     assert_equal(1, len(rows))
     assert_is_not_none(rows[0].values()[0])
 
-@parameterized(BOUNDARY_COLUMNS)
-def test_boundary_points(column_id):
-    resp = query('''
-SELECT * FROM {schema}OBS_GetBoundary({point}, '{column_id}')
-                 '''.format(column_id=column_id,
-                            schema='cdb_observatory.' if USE_SCHEMA else '',
-                            point=default_point(column_id)))
-    assert_equal(resp.status_code, 200)
-    rows = resp.json()['rows']
-    assert_equal(1, len(rows))
-    assert_is_not_none(rows[0].values()[0])
+#@parameterized(BOUNDARY_COLUMNS)
+#def test_get_boundaries_by_geometry(column_id):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetBoundariesByGeometry({area}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            area=default_area(column_id)))
+#    assert_equal(resp.status_code, 200)
+#    rows = resp.json()['rows']
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0].values()[0])
+
+#@parameterized(BOUNDARY_COLUMNS)
+#def test_get_points_by_geometry(column_id):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetPointsByGeometry({area}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            area=default_area(column_id)))
+#    assert_equal(resp.status_code, 200)
+#    rows = resp.json()['rows']
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0].values()[0])
+
+#@parameterized(BOUNDARY_COLUMNS)
+#def test_get_boundary_points(column_id):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetBoundary({point}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            point=default_point(column_id)))
+#    assert_equal(resp.status_code, 200)
+#    rows = resp.json()['rows']
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0].values()[0])
+
+#@parameterized(BOUNDARY_COLUMNS)
+#def test_get_boundary_id(column_id):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetBoundaryId({point}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            point=default_point(column_id)))
+#    assert_equal(resp.status_code, 200)
+#    rows = resp.json()['rows']
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0].values()[0])
+
+#@parameterized(BOUNDARY_COLUMNS)
+#def test_get_boundary_by_id(column_id):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetBoundaryById({geometry_id}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            geometry_id=default_geometry_id(column_id)))
+#    assert_equal(resp.status_code, 200)
+#    rows = resp.json()['rows']
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0].values()[0])

scripts/generate_fixtures.py

@@ -33,7 +33,8 @@ def select_star(tablename):
 
 cdb = Dumpr('observatory.cartodb.com','')
 
-metadata = ['obs_table', 'obs_column_table', 'obs_column', 'obs_column_tag', 'obs_tag', 'obs_column_to_column']
+metadata = ['obs_table', 'obs_column_table', 'obs_column', 'obs_column_tag',
+            'obs_tag', 'obs_column_to_column', 'obs_dump_version', ]
 
 fixtures = [
     ('us.census.tiger.census_tract', 'us.census.tiger.census_tract', '2014'),
@@ -46,6 +47,7 @@ fixtures = [
     ('us.census.spielman_singleton_segments.X10', 'us.census.tiger.census_tract', '2010 - 2014'),
     ('us.zillow.AllHomes_Zhvi', 'us.census.tiger.zcta5', '2014-01'),
     ('us.zillow.AllHomes_Zhvi', 'us.census.tiger.zcta5', '2016-03'),
+    ('whosonfirst.wof_country_geom', 'whosonfirst.wof_country_geom', '2016'),
     ('us.census.tiger.zcta5_clipped', 'us.census.tiger.zcta5_clipped', '2014'),
     ('us.census.tiger.block_group_clipped', 'us.census.tiger.block_group_clipped', '2014'),
 ]
@@ -76,10 +78,136 @@ with open('src/pg/test/fixtures/load_fixtures.sql', 'w') as outfile:
 
         for tablename, colname, boundary_id in unique_tables:
             if 'zcta5' in boundary_id:
-                where = '11%'
+                where = '\'11%\''
+                compare = 'LIKE'
+            elif 'whosonfirst' in boundary_id:
+                where = '(\'85632785\',\'85633051\',\'85633111\',\'85633147\',\'85633253\',\'85633267\')'
+                compare = 'IN'
             else:
-                where = '36047%'
-            print ' '.join([select_star(tablename), "WHERE {} LIKE '{}'".format(colname, where)])
-            cdb.dump(' '.join([select_star(tablename), "WHERE {} LIKE '{}'".format(colname, where)]),
-                    tablename, outfile, schema='observatory')
+                where = '\'36047%\''
+                compare = 'LIKE'
+            print ' '.join([select_star(tablename), "WHERE {}::text {} {}".format(colname, compare, where)])
+            cdb.dump(' '.join([select_star(tablename), "WHERE {}::text {} {}".format(colname, compare, where)]),
+                     tablename, outfile, schema='observatory')
             dropfiles.write('DROP TABLE IF EXISTS observatory.{};\n'.format(tablename))
+
+
+        outfile.write('''
+ALTER TABLE observatory.obs_table
+  ADD PRIMARY KEY (id);
+ALTER TABLE observatory.obs_column_table
+  ADD PRIMARY KEY (column_id, table_id);
+CREATE UNIQUE INDEX ON observatory.obs_column_table (table_id, column_id);
+CREATE UNIQUE INDEX ON observatory.obs_column_table (table_id, colname);
+ALTER TABLE observatory.obs_column
+  ADD PRIMARY KEY (id);
+ALTER TABLE observatory.obs_column_to_column
+  ADD PRIMARY KEY (source_id, target_id, reltype);
+CREATE UNIQUE INDEX ON observatory.obs_column_to_column (target_id, source_id, reltype);
+CREATE INDEX ON observatory.obs_column_to_column (reltype);
+ALTER TABLE observatory.obs_column_tag
+  ADD PRIMARY KEY (column_id, tag_id);
+CREATE UNIQUE INDEX ON observatory.obs_column_tag (tag_id, column_id);
+ALTER TABLE observatory.obs_tag
+  ADD PRIMARY KEY (id);
+CREATE INDEX ON observatory.obs_tag (type);
+
+VACUUM ANALYZE observatory.obs_table;
+VACUUM ANALYZE observatory.obs_column_table;
+VACUUM ANALYZE observatory.obs_column;
+VACUUM ANALYZE observatory.obs_column_to_column;
+VACUUM ANALYZE observatory.obs_column_tag;
+VACUUM ANALYZE observatory.obs_tag;
+
+CREATE TABLE observatory.obs_meta AS
+SELECT numer_c.id numer_id,
+       denom_c.id denom_id,
+       geom_c.id geom_id,
+       MAX(numer_c.name) numer_name,
+       MAX(denom_c.name) denom_name,
+       MAX(geom_c.name) geom_name,
+       MAX(numer_c.description) numer_description,
+       MAX(denom_c.description) denom_description,
+       MAX(geom_c.description) geom_description,
+       MAX(numer_c.aggregate) numer_aggregate,
+       MAX(denom_c.aggregate) denom_aggregate,
+       MAX(geom_c.aggregate) geom_aggregate,
+       MAX(numer_c.type) numer_type,
+       MAX(denom_c.type) denom_type,
+       MAX(geom_c.type) geom_type,
+       MAX(numer_data_ct.colname) numer_colname,
+       MAX(denom_data_ct.colname) denom_colname,
+       MAX(geom_geom_ct.colname) geom_colname,
+       MAX(numer_geomref_ct.colname) numer_geomref_colname,
+       MAX(denom_geomref_ct.colname) denom_geomref_colname,
+       MAX(geom_geomref_ct.colname) geom_geomref_colname,
+       MAX(numer_t.tablename) numer_tablename,
+       MAX(denom_t.tablename) denom_tablename,
+       MAX(geom_t.tablename) geom_tablename,
+       MAX(numer_t.timespan) numer_timespan,
+       MAX(denom_t.timespan) denom_timespan,
+       MAX(numer_c.weight) numer_weight,
+       MAX(denom_c.weight) denom_weight,
+       MAX(geom_c.weight) geom_weight,
+       MAX(geom_t.timespan) geom_timespan,
+       MAX(geom_t.the_geom_webmercator)::geometry AS the_geom_webmercator,
+       ARRAY_AGG(DISTINCT s_tag.id) section_tags,
+       ARRAY_AGG(DISTINCT ss_tag.id) subsection_tags,
+       ARRAY_AGG(DISTINCT unit_tag.id) unit_tags
+FROM observatory.obs_column_table numer_data_ct,
+     observatory.obs_table numer_t,
+     observatory.obs_column_table numer_geomref_ct,
+     observatory.obs_column geomref_c,
+     observatory.obs_column_to_column geomref_c2c,
+     observatory.obs_column geom_c,
+     observatory.obs_column_table geom_geom_ct,
+     observatory.obs_column_table geom_geomref_ct,
+     observatory.obs_table geom_t,
+     observatory.obs_column_tag ss_ctag,
+     observatory.obs_tag ss_tag,
+     observatory.obs_column_tag s_ctag,
+     observatory.obs_tag s_tag,
+     observatory.obs_column_tag unit_ctag,
+     observatory.obs_tag unit_tag,
+     observatory.obs_column numer_c
+  LEFT JOIN (
+    observatory.obs_column_to_column denom_c2c
+    JOIN observatory.obs_column denom_c ON denom_c2c.target_id = denom_c.id
+    JOIN observatory.obs_column_table denom_data_ct ON denom_data_ct.column_id = denom_c.id
+    JOIN observatory.obs_table denom_t ON denom_data_ct.table_id = denom_t.id
+    JOIN observatory.obs_column_table denom_geomref_ct ON denom_geomref_ct.table_id = denom_t.id
+  ) ON denom_c2c.source_id = numer_c.id
+WHERE numer_c.id = numer_data_ct.column_id
+  AND numer_data_ct.table_id = numer_t.id
+  AND numer_t.id = numer_geomref_ct.table_id
+  AND numer_geomref_ct.column_id = geomref_c.id
+  AND geomref_c2c.reltype = 'geom_ref'
+  AND geomref_c.id = geomref_c2c.source_id
+  AND geom_c.id = geomref_c2c.target_id
+  AND geom_geomref_ct.column_id = geomref_c.id
+  AND geom_geomref_ct.table_id = geom_t.id
+  AND geom_geom_ct.column_id = geom_c.id
+  AND geom_geom_ct.table_id = geom_t.id
+  AND geom_c.type ILIKE 'geometry'
+  AND numer_c.type NOT ILIKE 'geometry'
+  AND numer_t.id != geom_t.id
+  AND numer_c.id != geomref_c.id
+  AND unit_tag.type = 'unit'
+  AND ss_tag.type = 'subsection'
+  AND s_tag.type = 'section'
+  AND unit_ctag.column_id = numer_c.id
+  AND unit_ctag.tag_id = unit_tag.id
+  AND ss_ctag.column_id = numer_c.id
+  AND ss_ctag.tag_id = ss_tag.id
+  AND s_ctag.column_id = numer_c.id
+  AND s_ctag.tag_id = s_tag.id
+  AND (denom_c2c.reltype = 'denominator' OR denom_c2c.reltype IS NULL)
+  AND (denom_geomref_ct.column_id = geomref_c.id OR denom_geomref_ct.column_id IS NULL)
+  AND (denom_t.timespan = numer_t.timespan OR denom_t.timespan IS NULL)
+GROUP BY numer_c.id, denom_c.id, geom_c.id,
+         numer_t.id, denom_t.id, geom_t.id;
+        ''')
+
+        dropfiles.write('''
+DROP TABLE IF EXISTS observatory.obs_meta;
+                        ''')

src/pg/observatory.control

@@ -1,5 +1,5 @@
 comment = 'CartoDB Observatory backend extension'
-default_version = '0.0.2'
+default_version = '1.0.2'
 requires = 'postgis'
 superuser = true
 schema = cdb_observatory

src/pg/sql/15_fdw_utilities.sql

@@ -0,0 +1,67 @@
+CREATE OR REPLACE FUNCTION cdb_observatory._OBS_ConnectRemoteTable(fdw_name text, schema_name text, user_dbname text, user_hostname text, username text, user_tablename text, user_schema text)
+RETURNS void
+AS $$
+DECLARE
+  row record;
+  option record;
+  connection_str json;
+BEGIN
+  -- Build connection string
+  connection_str := '{"server":{"extensions":"postgis", "dbname":"'
+    || user_dbname ||'", "host":"' || user_hostname ||'", "port":"5432"}, "users":{"public"'
+    || ':{"user":"' || username ||'", "password":""} } }';
+
+  -- This function tries to be as idempotent as possible, by not creating anything more than once
+  -- (not even using IF NOT EXIST to avoid throwing warnings)
+  IF NOT EXISTS ( SELECT * FROM pg_extension WHERE extname = 'postgres_fdw') THEN
+    CREATE EXTENSION postgres_fdw;
+  END IF;
+  -- Create FDW first if it does not exist
+  IF NOT EXISTS ( SELECT * FROM pg_foreign_server WHERE srvname = fdw_name)
+    THEN
+    EXECUTE FORMAT('CREATE SERVER %I FOREIGN DATA WRAPPER postgres_fdw', fdw_name);
+  END IF;
+
+  -- Set FDW settings
+  FOR row IN SELECT p.key, p.value from lateral json_each_text(connection_str->'server') p
+    LOOP
+      IF NOT EXISTS (WITH a AS (select split_part(unnest(srvoptions), '=', 1) as options from pg_foreign_server where srvname=fdw_name) SELECT * from a where options = row.key)
+        THEN
+        EXECUTE FORMAT('ALTER SERVER %I OPTIONS (ADD %I %L)', fdw_name, row.key, row.value);
+      ELSE
+        EXECUTE FORMAT('ALTER SERVER %I OPTIONS (SET %I %L)', fdw_name, row.key, row.value);
+      END IF;
+    END LOOP;
+
+    -- Create user mappings
+    FOR row IN SELECT p.key, p.value from lateral json_each(connection_str->'users') p LOOP
+      -- Check if entry on pg_user_mappings exists
+      IF NOT EXISTS ( SELECT * FROM pg_user_mappings WHERE srvname = fdw_name AND usename = row.key ) THEN
+        EXECUTE FORMAT ('CREATE USER MAPPING FOR %I SERVER %I', row.key, fdw_name);
+      END IF;
+
+      -- Update user mapping settings
+      FOR option IN SELECT o.key, o.value from lateral json_each_text(row.value) o LOOP
+        IF NOT EXISTS (WITH a AS (select split_part(unnest(umoptions), '=', 1) as options from pg_user_mappings WHERE srvname = fdw_name AND usename = row.key) SELECT * from a where options = option.key) THEN
+          EXECUTE FORMAT('ALTER USER MAPPING FOR %I SERVER %I OPTIONS (ADD %I %L)', row.key, fdw_name, option.key, option.value);
+        ELSE
+          EXECUTE FORMAT('ALTER USER MAPPING FOR %I SERVER %I OPTIONS (SET %I %L)', row.key, fdw_name, option.key, option.value);
+        END IF;
+      END LOOP;
+    END LOOP;
+
+    -- Create schema if it does not exist.
+    IF NOT EXISTS ( SELECT * from pg_namespace WHERE nspname=fdw_name) THEN
+      EXECUTE FORMAT ('CREATE SCHEMA %I', fdw_name);
+    END IF;
+
+    -- Bring the remote cdb_tablemetadata
+    IF NOT EXISTS ( SELECT * FROM PG_CLASS WHERE relnamespace = (SELECT oid FROM pg_namespace WHERE nspname=fdw_name) and relname='cdb_tablemetadata') THEN
+      EXECUTE FORMAT ('CREATE FOREIGN TABLE %I.cdb_tablemetadata (tabname text, updated_at timestamp with time zone) SERVER %I OPTIONS (table_name ''cdb_tablemetadata_text'', schema_name ''public'', updatable ''false'')', fdw_name, fdw_name);
+    END IF;
+
+    -- Import target table
+    EXECUTE FORMAT ('IMPORT FOREIGN SCHEMA %I LIMIT TO (%I) from SERVER %I INTO %I', user_schema, user_tablename, fdw_name, schema_name);
+
+END;
+$$ LANGUAGE PLPGSQL;

src/pg/sql/40_observatory_utility.sql

@@ -158,28 +158,6 @@ BEGIN
 END;
 $$ LANGUAGE plpgsql;
 
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetRelatedColumn(columns_ids text[], reltype text )
-RETURNS TEXT[]
-AS $$
-DECLARE
-  result TEXT[];
-BEGIN
-  EXECUTE '
-    With ids as (
-      select row_number() over() as no, id from (select unnest($1) as id) t
-    )
-    select array_agg(target_id order by no)
-    FROM  ids
-    LEFT JOIN observatory.obs_column_to_column
-    on  source_id  = id
-    where reltype = $2 or reltype is null
-  '
-  INTO result
-  using columns_ids, reltype;
-  return result;
-END;
-$$ LANGUAGE plpgsql;
-
 -- Function that replaces all non digits or letters with _ trims and lowercases the
 -- passed measure name
 
@@ -199,3 +177,20 @@ BEGIN
   RETURN result;
 END;
 $$ LANGUAGE plpgsql;
+
+-- Function that returns the currently deployed obs_dump_version from the
+-- remote table of the same name.
+
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_DumpVersion(
+)
+  RETURNS TEXT
+AS $$
+DECLARE
+  result text;
+BEGIN
+  EXECUTE '
+    SELECT MAX(dump_id) FROM observatory.obs_dump_version
+  ' INTO result;
+  RETURN result;
+END;
+$$ LANGUAGE plpgsql;

src/pg/sql/41_observatory_augmentation.sql

@@ -169,19 +169,19 @@ BEGIN
 
   IF geom_table_name IS NULL
   THEN
-     RAISE NOTICE 'Point % is outside of the data region', ST_AsText(geom);
+     --raise notice 'Point % is outside of the data region', ST_AsText(geom);
       -- TODO this should return JSON
      RETURN QUERY SELECT '{}'::json;
      RETURN;
   END IF;
 
   IF data_table_info IS NULL THEN
-    RAISE NOTICE 'Cannot find data table for boundary ID %, column_ids %, and time_span %', geometry_level, column_ids, time_span;
+    --raise notice 'Cannot find data table for boundary ID %, column_ids %, and time_span %', geometry_level, column_ids, time_span;
   END IF;
 
   IF ST_GeometryType(geom) = 'ST_Point'
   THEN
-    RAISE NOTICE 'geom_table_name %, data_table_info %', geom_table_name, data_table_info::json[];
+    --raise notice 'geom_table_name %, data_table_info %', geom_table_name, data_table_info::json[];
     results := cdb_observatory._OBS_GetPoints(geom,
                                               geom_table_name,
                                               data_table_info);
@@ -260,7 +260,7 @@ BEGIN
   USING geom
   INTO geoid;
 
-  RAISE NOTICE 'geoid is %, geometry table is % ', geoid, geom_table_name;
+  --raise notice 'geoid is %, geometry table is % ', geoid, geom_table_name;
 
   EXECUTE
     format('SELECT ST_Area(the_geom::geography) / (1000 * 1000)
@@ -273,7 +273,7 @@ BEGIN
 
   IF area IS NULL
   THEN
-    RAISE NOTICE 'No geometry at %', ST_AsText(geom);
+    --raise notice 'No geometry at %', ST_AsText(geom);
   END IF;
 
   query := 'SELECT Array[';
@@ -338,48 +338,221 @@ $$ LANGUAGE plpgsql;
 CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetMeasure(
   geom geometry(Geometry, 4326),
   measure_id TEXT,
-  normalize TEXT DEFAULT 'area', -- TODO none/null
+  normalize TEXT DEFAULT NULL,
   boundary_id TEXT DEFAULT NULL,
   time_span TEXT DEFAULT NULL
 )
 RETURNS NUMERIC
 AS $$
 DECLARE
+  geom_type TEXT;
+  map_type TEXT;
+  numer_aggregate TEXT;
+  numer_colname TEXT;
+  numer_geomref_colname TEXT;
+  numer_tablename TEXT;
+  denom_colname TEXT;
+  denom_geomref_colname TEXT;
+  denom_tablename TEXT;
+  geom_colname TEXT;
+  geom_geomref_colname TEXT;
+  geom_tablename TEXT;
   result NUMERIC;
-  measure_ids TEXT[];
-  denominator_id TEXT;
-  vals NUMERIC[];
+  sql TEXT;
+  numer_name TEXT;
 BEGIN
 
-  IF normalize ILIKE 'area' THEN
-    measure_ids := ARRAY[measure_id];
+  EXECUTE
+     $query$
+     SELECT numer_aggregate, numer_colname, numer_geomref_colname, numer_tablename,
+            denom_colname, denom_geomref_colname, denom_tablename,
+            geom_colname, geom_geomref_colname, geom_tablename, numer_name
+             FROM observatory.obs_meta
+             WHERE (geom_id = $1 OR ($1 = ''))
+               AND numer_id = $2
+               AND (numer_timespan = $3 OR ($3 = ''))
+             ORDER BY geom_weight DESC, numer_timespan DESC
+             LIMIT 1
+     $query$
+    INTO numer_aggregate, numer_colname, numer_geomref_colname, numer_tablename,
+         denom_colname, denom_geomref_colname, denom_tablename,
+         geom_colname, geom_geomref_colname, geom_tablename, numer_name
+    USING COALESCE(boundary_id, ''), measure_id, COALESCE(time_span, '');
+
+  IF ST_GeometryType(geom) = 'ST_Point' THEN
+    geom_type := 'point';
+  ELSIF ST_GeometryType(geom) IN ('ST_Polygon', 'ST_MultiPolygon') THEN
+    geom_type := 'polygon';
+  ELSE
+    RAISE EXCEPTION 'Invalid geometry type (%), can only handle ''ST_Point'', ''ST_Polygon'', and ''ST_MultiPolygon''',
+                    ST_GeometryType(geom);
+  END IF;
+
+  IF normalize ILIKE 'area' AND numer_aggregate ILIKE 'sum' THEN
+    map_type := 'areaNormalized';
   ELSIF normalize ILIKE 'denominator' THEN
-    EXECUTE 'SELECT (cdb_observatory._OBS_GetRelatedColumn(ARRAY[$1], ''denominator''))[1]
-    ' INTO denominator_id
-    USING measure_id;
-    measure_ids := ARRAY[measure_id, denominator_id];
-  ELSIF normalize ILIKE 'none' THEN
-    -- TODO we need a switch on obs_get to disable area normalization
-    RAISE EXCEPTION 'No normalization not yet supported.';
+    map_type := 'denominated';
   ELSE
-    RAISE EXCEPTION 'Only valid inputs for "normalize" are "area" (default) and "denominator".';
+    -- defaults: area normalization for point if it's possible and none for
+    -- polygon or non-summable point
+    IF geom_type = 'point' AND numer_aggregate ILIKE 'sum' THEN
+      map_type := 'areaNormalized';
+    ELSE
+      map_type := 'predenominated';
+    END IF;
   END IF;
 
-  EXECUTE '
-    SELECT ARRAY_AGG(val) FROM (SELECT (cdb_observatory._OBS_Get($1, $2, $3, $4)->>''value'')::NUMERIC val) b
-  '
-  INTO vals
-  USING geom, measure_ids, time_span, boundary_id;
-
-  IF normalize ILIKE 'denominator' THEN
-    RETURN (vals)[1]/(vals)[2];
-  ELSE
-    RETURN (vals)[1];
+  IF geom_type = 'point' THEN
+    IF map_type = 'areaNormalized' THEN
+      sql = format('WITH _geom AS (SELECT ST_Area(geom.%I::Geography) / 1000000 area, geom.%I geom_ref
+                                   FROM observatory.%I geom
+                                   WHERE ST_Within(%L, geom.%I)
+                                   LIMIT 1)
+                    SELECT numer.%I / (SELECT area FROM _geom)
+                    FROM observatory.%I numer
+                    WHERE numer.%I = (SELECT geom_ref FROM _geom)',
+                geom_colname, geom_geomref_colname, geom_tablename,
+                geom, geom_colname, numer_colname, numer_tablename,
+                numer_geomref_colname);
+    ELSIF map_type = 'denominated' THEN
+      sql = format('SELECT numer.%I / NULLIF((SELECT denom.%I FROM observatory.%I denom WHERE denom.%I = numer.%I LIMIT 1), 0)
+                    FROM observatory.%I numer
+                    WHERE numer.%I = (SELECT geom.%I FROM observatory.%I geom WHERE ST_Within(%L, geom.%I) LIMIT 1)',
+                        numer_colname, denom_colname, denom_tablename,
+                        denom_geomref_colname, numer_geomref_colname,
+                        numer_tablename,
+                        numer_geomref_colname, geom_geomref_colname,
+                        geom_tablename, geom, geom_colname);
+    ELSIF map_type = 'predenominated' THEN
+      sql = format('SELECT numer.%I
+                    FROM observatory.%I numer
+                    WHERE numer.%I = (SELECT geom.%I FROM observatory.%I geom WHERE ST_Within(%L, geom.%I) LIMIT 1)',
+                        numer_colname, numer_tablename,
+                        numer_geomref_colname, geom_geomref_colname, geom_tablename,
+                        geom, geom_colname);
+    END IF;
+  ELSIF geom_type = 'polygon' THEN
+    IF map_type = 'areaNormalized' THEN
+      sql = format('WITH _geom AS (SELECT ST_Area(ST_Intersection(%L, geom.%I))
+                                        / ST_Area(geom.%I) overlap, geom.%I geom_ref
+                                   FROM observatory.%I geom
+                                   WHERE ST_Intersects(%L, geom.%I)
+                                     AND ST_Area(ST_Intersection(%L, geom.%I)) / ST_Area(geom.%I) > 0)
+                    SELECT SUM(numer.%I * (SELECT _geom.overlap FROM _geom WHERE _geom.geom_ref = numer.%I)) /
+                           ST_Area(%L::Geography)
+                    FROM observatory.%I numer
+                    WHERE numer.%I = ANY ((SELECT ARRAY_AGG(geom_ref) FROM _geom)::TEXT[])',
+                geom, geom_colname, geom_colname,
+                geom_geomref_colname, geom_tablename,
+                geom, geom_colname,
+                geom, geom_colname, geom_colname,
+                numer_colname, numer_geomref_colname,
+                geom, numer_tablename,
+                numer_geomref_colname);
+    ELSIF map_type = 'denominated' THEN
+      sql = format('WITH _geom AS (SELECT ST_Area(ST_Intersection(%L, geom.%I))
+                                        / ST_Area(geom.%I) overlap, geom.%I geom_ref
+                                   FROM observatory.%I geom
+                                   WHERE ST_Intersects(%L, geom.%I)
+                                     AND ST_Area(ST_Intersection(%L, geom.%I)) / ST_Area(geom.%I) > 0),
+                        _denom AS (SELECT denom.%I, denom.%I geom_ref
+                                   FROM observatory.%I denom
+                                   WHERE denom.%I = ANY ((SELECT ARRAY_AGG(geom_ref) FROM _geom)::TEXT[]))
+                    SELECT SUM(numer.%I * (SELECT _geom.overlap FROM _geom WHERE _geom.geom_ref = numer.%I)) /
+                           SUM((SELECT _denom.%I * (SELECT _geom.overlap
+                                                    FROM _geom
+                                                    WHERE _geom.geom_ref = _denom.geom_ref)
+                                FROM _denom WHERE _denom.geom_ref = numer.%I))
+                    FROM observatory.%I numer
+                    WHERE numer.%I = ANY ((SELECT ARRAY_AGG(geom_ref) FROM _geom)::TEXT[])',
+                geom, geom_colname,
+                geom_colname, geom_geomref_colname,
+                geom_tablename,
+                geom, geom_colname,
+                geom, geom_colname, geom_colname,
+                denom_colname, denom_geomref_colname,
+                denom_tablename,
+                denom_geomref_colname,
+                numer_colname, numer_geomref_colname,
+                denom_colname,
+                numer_geomref_colname,
+                numer_tablename,
+                numer_geomref_colname);
+    ELSIF map_type = 'predenominated' THEN
+      IF numer_aggregate NOT ILIKE 'sum' THEN
+        RAISE EXCEPTION 'Cannot calculate "%" (%) for custom area as it cannot be summed, use ST_PointOnSurface instead',
+                        numer_name, numer_id;
+      ELSE
+        sql = format('WITH _geom AS (SELECT ST_Area(ST_Intersection(%L, geom.%I))
+                                          / ST_Area(geom.%I) overlap, geom.%I geom_ref
+                                     FROM observatory.%I geom
+                                     WHERE ST_Intersects(%L, geom.%I)
+                                       AND ST_Area(ST_Intersection(%L, geom.%I)) / ST_Area(geom.%I) > 0)
+                      SELECT SUM(numer.%I * (SELECT _geom.overlap FROM _geom WHERE _geom.geom_ref = numer.%I))
+                      FROM observatory.%I numer
+                      WHERE numer.%I = ANY ((SELECT ARRAY_AGG(geom_ref) FROM _geom)::TEXT[])',
+                  geom, geom_colname, geom_colname,
+                  geom_geomref_colname, geom_tablename,
+                  geom, geom_colname,
+                  geom, geom_colname, geom_colname,
+                  numer_colname, numer_geomref_colname,
+                  numer_tablename,
+                  numer_geomref_colname);
+      END IF;
+    END IF;
   END IF;
+
+  EXECUTE sql INTO result;
+  RETURN result;
+
 END;
 $$ LANGUAGE plpgsql;
 
 
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetMeasureById(
+  geom_ref TEXT,
+  measure_id TEXT,
+  boundary_id TEXT,
+  time_span TEXT DEFAULT NULL
+)
+RETURNS NUMERIC
+AS $$
+DECLARE
+  target_table TEXT;
+  colname TEXT;
+  measure_val NUMERIC;
+  data_geoid_colname TEXT;
+BEGIN
+
+  EXECUTE
+     $query$
+     SELECT numer_colname, numer_geomref_colname, numer_tablename
+             FROM observatory.obs_meta
+             WHERE (geom_id = $1 OR ($1 = ''))
+               AND numer_id = $2
+               AND (numer_timespan = $3 OR ($3 = ''))
+             ORDER BY geom_weight DESC, numer_timespan DESC
+             LIMIT 1
+     $query$
+    INTO colname, data_geoid_colname, target_table
+    USING COALESCE(boundary_id, ''), measure_id, COALESCE(time_span, '');
+
+  --RAISE DEBUG 'target_table %, colname %', target_table, colname;
+
+  EXECUTE format(
+      'SELECT %I
+       FROM observatory.%I data
+       WHERE data.%I = %L',
+       colname,
+       target_table,
+       data_geoid_colname, geom_ref)
+  INTO measure_val;
+
+  RETURN measure_val;
+
+END;
+$$ LANGUAGE plpgsql;
+
 CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetCategory(
   geom geometry(Geometry, 4326),
   category_id TEXT,
@@ -389,27 +562,61 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetCategory(
 RETURNS TEXT
 AS $$
 DECLARE
-  denominator_id TEXT;
-  categories TEXT[];
+  data_table TEXT;
+  geom_table TEXT;
+  colname TEXT;
+  data_geomref_colname TEXT;
+  geom_geomref_colname TEXT;
+  geom_colname TEXT;
+  category_val TEXT;
+  category_share NUMERIC;
 BEGIN
 
-  IF boundary_id IS NULL THEN
-    -- TODO we should determine best boundary for this geom
-    boundary_id := 'us.census.tiger.census_tract';
+  EXECUTE
+     $query$
+     SELECT numer_colname, numer_geomref_colname, numer_tablename,
+            geom_geomref_colname, geom_colname, geom_tablename
+             FROM observatory.obs_meta
+             WHERE (geom_id = $1 OR ($1 = ''))
+               AND numer_id = $2
+               AND (numer_timespan = $3 OR ($3 = ''))
+             ORDER BY geom_weight DESC, numer_timespan DESC
+             LIMIT 1
+     $query$
+    INTO colname, data_geomref_colname, data_table,
+         geom_geomref_colname, geom_colname, geom_table
+    USING COALESCE(boundary_id, ''), category_id, COALESCE(time_span, '');
+
+  IF ST_GeometryType(geom) = 'ST_Point' THEN
+    EXECUTE format(
+        'SELECT data.%I
+         FROM observatory.%I data, observatory.%I geom
+         WHERE data.%I = geom.%I
+           AND ST_WITHIN(%L, geom.%I) ',
+         colname, data_table, geom_table, data_geomref_colname,
+         geom_geomref_colname, geom, geom_colname)
+    INTO category_val;
+  ELSE
+    -- favor the category with the most area
+    EXECUTE format(
+       'SELECT data.%I category, SUM(overlap_fraction) category_share
+        FROM observatory.%I data, (
+          SELECT ST_Area(
+           ST_Intersection(%L, a.%I)
+          ) / ST_Area(%L) AS overlap_fraction, a.%I geomref
+          FROM observatory.%I as a
+          WHERE %L && a.%I) _overlaps
+        WHERE data.%I = _overlaps.geomref
+        GROUP BY category
+        ORDER BY SUM(overlap_fraction) DESC
+        LIMIT 1',
+          colname, data_table,
+          geom, geom_colname, geom, geom_geomref_colname,
+          geom_table, geom, geom_colname, data_geomref_colname)
+    INTO category_val, category_share;
   END IF;
 
-  IF time_span IS NULL THEN
-    -- TODO we should determine latest timespan for this measure
-    time_span := '2010 - 2014';
-  END IF;
-
-  EXECUTE '
-    SELECT ARRAY_AGG(val) FROM (SELECT (cdb_observatory._OBS_GetCategories($1, $2, $3, $4))->>''category'' val LIMIT 1) b
-  '
-  INTO categories
-  USING geom, ARRAY[category_id], boundary_id, time_span;
-
-  RETURN (categories)[1];
+  RETURN category_val;
 
 END;
 $$ LANGUAGE plpgsql;
@@ -417,7 +624,7 @@ $$ LANGUAGE plpgsql;
 CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetUSCensusMeasure(
    geom geometry(Geometry, 4326),
    name TEXT,
-   normalize TEXT DEFAULT 'area',
+   normalize TEXT DEFAULT NULL,
    boundary_id TEXT DEFAULT NULL,
    time_span TEXT DEFAULT NULL
  )
@@ -435,7 +642,7 @@ BEGIN
        JOIN observatory.obs_column_tag ct
          ON c.id = ct.column_id
        WHERE cdb_observatory._OBS_StandardizeMeasureName(c.name) = $1
-         AND ct.tag_id = 'us.census.acs.demographics'
+         AND ct.tag_id ILIKE 'us.census%'
   $string$
   INTO measure_id
   USING standardized_name;
@@ -483,7 +690,7 @@ $$ LANGUAGE plpgsql;
 
 CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetPopulation(
   geom geometry(Geometry, 4326),
-  normalize TEXT DEFAULT 'area',
+  normalize TEXT DEFAULT NULL,
   boundary_id TEXT DEFAULT NULL,
   time_span TEXT DEFAULT NULL
 )
@@ -520,9 +727,37 @@ DECLARE
   q_sum text;
   q text;
   i NUMERIC;
+  data_geoid_colname text;
+  geom_geoid_colname text;
 BEGIN
 
-  q_select := 'SELECT geoid, ';
+  -- TODO we're assuming our geom_table has only one geom_ref column
+  --      we *really* should pass in both geom_table_name and boundary_id
+  -- TODO tablename should not be passed here (use boundary_id)
+  EXECUTE
+    format('SELECT ct.colname
+              FROM observatory.obs_column_to_column c2c,
+                   observatory.obs_column_table ct,
+                   observatory.obs_table t
+             WHERE c2c.reltype = ''geom_ref''
+               AND ct.column_id = c2c.source_id
+               AND ct.table_id = t.id
+               AND t.tablename = %L'
+     , (data_table_info)[1]->>'tablename')
+  INTO data_geoid_colname;
+  EXECUTE
+    format('SELECT ct.colname
+              FROM observatory.obs_column_to_column c2c,
+                   observatory.obs_column_table ct,
+                   observatory.obs_table t
+             WHERE c2c.reltype = ''geom_ref''
+               AND ct.column_id = c2c.source_id
+               AND ct.table_id = t.id
+               AND t.tablename = %L'
+   , geom_table_name)
+  INTO geom_geoid_colname;
+
+  q_select := format('SELECT %I, ', data_geoid_colname);
   q_sum    := 'SELECT Array[';
 
   FOR i IN 1..array_upper(data_table_info, 1)
@@ -543,22 +778,22 @@ BEGIN
     END IF;
  END LOOP;
 
-  q = format('
+  q := format('
     WITH _overlaps As (
       SELECT ST_Area(
         ST_Intersection($1, a.the_geom)
       ) / ST_Area(a.the_geom) As overlap_fraction,
-      geoid
+      %I
       FROM observatory.%I As a
       WHERE $1 && a.the_geom
     ),
     values As (
-    ', geom_table_name);
+    ', geom_geoid_colname, geom_table_name);
 
   q := q || q_select || format('FROM observatory.%I ', ((data_table_info)[1]->>'tablename'));
 
-  q := q || ' ) ' || q_sum || ' ]::numeric[] FROM _overlaps, values
-  WHERE values.geoid = _overlaps.geoid';
+  q := format(q || ' ) ' || q_sum || ' ]::numeric[] FROM _overlaps, values
+  WHERE values.%I = _overlaps.%I', data_geoid_colname, geom_geoid_colname);
 
   EXECUTE
     q
@@ -721,7 +956,7 @@ BEGIN
 
   IF geom_table_name IS NULL
   THEN
-     RAISE NOTICE 'Point % is outside of the data region', ST_AsText(geom);
+     --raise notice 'Point % is outside of the data region', ST_AsText(geom);
      RETURN QUERY SELECT '{}'::text[], '{}'::text[];
      RETURN;
   END IF;
@@ -735,7 +970,7 @@ BEGIN
 
   IF data_table_info IS NULL
   THEN
-    RAISE NOTICE 'No data table found for this location';
+    --raise notice 'No data table found for this location';
     RETURN QUERY SELECT NULL::json;
     RETURN;
   END IF;
@@ -750,7 +985,7 @@ BEGIN
 
   IF geoid IS NULL
   THEN
-    RAISE NOTICE 'No geometry id for this location';
+    --raise notice 'No geometry id for this location';
     RETURN QUERY SELECT NULL::json;
     RETURN;
   END IF;

src/pg/sql/42_observatory_exploration.sql

@@ -114,7 +114,7 @@ BEGIN
       AND
         observatory.OBS_column.type = 'Geometry'
       AND
-        $1 && bounds::box2d
+        ST_Intersects($1, st_setsrid(observatory.obs_table.the_geom, 4326))
   $string$ || timespan_query
   USING geom;
   RETURN;

src/pg/sql/44_observatory_geometries.sql

@@ -64,11 +64,11 @@ BEGIN
   -- if no tables are found, raise notice and return null
   IF target_table IS NULL
   THEN
-    RAISE NOTICE 'No boundaries found for ''%'' in ''%''', ST_AsText(geom), boundary_id;
+    --RAISE NOTICE 'No boundaries found for ''%'' in ''%''', ST_AsText(geom), boundary_id;
     RETURN NULL::geometry;
   END IF;
 
-  RAISE NOTICE 'target_table: %', target_table;
+  --RAISE NOTICE 'target_table: %', target_table;
 
   -- return the first boundary in intersections
   EXECUTE format(
@@ -143,7 +143,7 @@ BEGIN
   -- if no tables are found, raise notice and return null
   IF target_table IS NULL
   THEN
-    RAISE NOTICE 'Warning: No boundaries found for ''%''', boundary_id;
+    --RAISE NOTICE 'Warning: No boundaries found for ''%''', boundary_id;
     RETURN NULL::text;
   END IF;
 
@@ -159,11 +159,11 @@ BEGIN
    , target_table)
   INTO geoid_colname;
 
-  RAISE NOTICE 'target_table: %, geoid_colname: %', target_table, geoid_colname;
+  --RAISE NOTICE 'target_table: %, geoid_colname: %', target_table, geoid_colname;
 
-  -- return name of geometry id column
+  -- return geometry id column value
   EXECUTE format(
-    'SELECT %I
+    'SELECT %I::text
      FROM observatory.%I
      WHERE ST_Intersects($1, the_geom)
      LIMIT 1', geoid_colname, target_table)
@@ -212,11 +212,11 @@ BEGIN
   SELECT * INTO geoid_colname, target_table, geom_colname
   FROM cdb_observatory._OBS_GetGeometryMetadata(boundary_id);
 
-  RAISE NOTICE '%', target_table;
+  --RAISE NOTICE '%', target_table;
 
   IF target_table IS NULL
   THEN
-    RAISE NOTICE 'No geometries found';
+    --RAISE NOTICE 'No geometries found';
     RETURN NULL::geometry;
   END IF;
 
@@ -272,17 +272,17 @@ BEGIN
   -- if no tables are found, raise notice and return null
   IF target_table IS NULL
   THEN
-    RAISE NOTICE 'No boundaries found for bounding box ''%'' in ''%''', ST_AsText(geom), boundary_id;
+    --RAISE NOTICE 'No boundaries found for bounding box ''%'' in ''%''', ST_AsText(geom), boundary_id;
     RETURN QUERY SELECT NULL::geometry, NULL::text;
     RETURN;
   END IF;
 
-  RAISE NOTICE 'target_table: %', target_table;
+  --RAISE NOTICE 'target_table: %', target_table;
 
   -- return first boundary in intersections
   RETURN QUERY
   EXECUTE format(
-    'SELECT %I, %I
+    'SELECT %I, %I::text
      FROM observatory.%I
      WHERE ST_%s($1, the_geom)
      ', geom_colname, geoid_colname, target_table, overlap_type)
@@ -409,7 +409,7 @@ BEGIN
     RAISE EXCEPTION 'Overlap type ''%'' is not an accepted type (choose intersects, within, or contains)', overlap_type;
   ELSIF ST_GeometryType(geom) NOT IN ('ST_Polygon', 'ST_MultiPolygon')
   THEN
-      RAISE EXCEPTION 'Invalid geometry type (%), expecting ''ST_MultiPolygon'' or ''ST_Polygon''', ST_GeometryType(geom);
+    RAISE EXCEPTION 'Invalid geometry type (%), expecting ''ST_MultiPolygon'' or ''ST_Polygon''', ST_GeometryType(geom);
   END IF;
 
   SELECT * INTO geoid_colname, target_table, geom_colname
@@ -418,17 +418,17 @@ BEGIN
   -- if no tables are found, raise notice and return null
   IF target_table IS NULL
   THEN
-    RAISE NOTICE 'No boundaries found for bounding box ''%'' in ''%''', ST_AsText(geom), boundary_id;
+    --RAISE NOTICE 'No boundaries found for bounding box ''%'' in ''%''', ST_AsText(geom), boundary_id;
     RETURN QUERY SELECT NULL::geometry, NULL::text;
     RETURN;
   END IF;
 
-  RAISE NOTICE 'target_table: %', target_table;
+  --RAISE NOTICE 'target_table: %', target_table;
 
   -- return first boundary in intersections
   RETURN QUERY
   EXECUTE format(
-    'SELECT ST_PointOnSurface(%I) As %s, %I
+    'SELECT ST_PointOnSurface(%I) As %s, %I::text
      FROM observatory.%I
      WHERE ST_%s($1, the_geom)
      ', geom_colname, geom_colname, geoid_colname, target_table, overlap_type)
@@ -565,6 +565,7 @@ BEGIN
           geom_c.type ILIKE 'geometry' AND
           geom_c.id = '%s'
     $string$, boundary_id, boundary_id);
+  RETURN;
     --  AND geom_t.timespan = '%s' <-- put in requested year
     -- TODO: filter by clipped vs. not so appropriate tablename are unique
     --       so the limit 1 can be removed

src/pg/sql/50_table_level_functions.sql

@@ -0,0 +1,119 @@
+CREATE TYPE cdb_observatory.ds_fdw_metadata as (schemaname text, tabname text, servername text);
+CREATE TYPE cdb_observatory.ds_return_metadata as (colnames text[], coltypes text[]);
+
+CREATE OR REPLACE FUNCTION cdb_observatory._OBS_ConnectUserTable(username text, orgname text, user_db_role text, input_schema text, dbname text, host_addr text, table_name text)
+RETURNS cdb_observatory.ds_fdw_metadata
+AS $$
+DECLARE
+  fdw_server text;
+  fdw_import_schema text;
+  connection_str json;
+  import_foreign_schema_q text;
+  epoch_timestamp text;
+BEGIN
+
+  SELECT extract(epoch from now() at time zone 'utc')::int INTO epoch_timestamp;
+  fdw_server := 'fdw_server_' || username || '_' || epoch_timestamp;
+  fdw_import_schema:= fdw_server;
+
+  -- Import foreign table
+  EXECUTE FORMAT ('SELECT cdb_observatory._OBS_ConnectRemoteTable(%L, %L, %L, %L, %L, %L, %L)', fdw_server, fdw_import_schema, dbname, host_addr, user_db_role, table_name, input_schema);
+
+  RETURN (fdw_import_schema::text, table_name::text, fdw_server::text);
+
+EXCEPTION
+  WHEN others THEN
+    -- Disconnect user imported table. Delete schema and FDW server.
+    EXECUTE 'DROP FOREIGN TABLE IF EXISTS ' || fdw_import_schema || '.' || table_name;
+    EXECUTE 'DROP SCHEMA IF EXISTS ' || fdw_import_schema || ' CASCADE';
+    EXECUTE 'DROP SERVER IF EXISTS ' || fdw_server || ' CASCADE;';
+    RETURN (null, null, null);
+END;
+$$ LANGUAGE plpgsql SECURITY DEFINER;
+
+CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetReturnMetadata(username text, orgname text, function_name text, params json)
+RETURNS cdb_observatory.ds_return_metadata
+AS $$
+DECLARE
+  colnames text[];
+  coltypes text[];
+  requested_measures text[];
+  measure text;
+BEGIN
+
+  -- Simple mock, there should be real logic in here.
+
+  IF $3 NOT ILIKE 'GetMeasure' OR $3 IS NULL THEN
+    RAISE 'This function is not supported yet: %', $3;
+  END IF;
+
+  SELECT translate($4::json->>'tag_name','[]', '{}')::text[] INTO requested_measures;
+
+  FOREACH measure IN ARRAY requested_measures
+  LOOP
+    IF NOT measure ILIKE ANY (Array['total_pop', 'pop_16_over']::text[]) THEN
+      RAISE 'This measure is not supported yet: %', measure;
+    END IF;
+  SELECT array_append(colnames, measure) INTO colnames;
+  SELECT array_append(coltypes, 'double precision'::text) INTO coltypes;
+
+  END LOOP;
+
+  RETURN (colnames::text[], coltypes::text[]);
+END;
+$$ LANGUAGE plpgsql;
+
+CREATE OR REPLACE FUNCTION cdb_observatory._OBS_FetchJoinFdwTableData(username text, orgname text, table_schema text, table_name text, function_name text, params json)
+RETURNS SETOF record
+AS $$
+DECLARE
+  data_query text;
+  tag_name text[];
+  tag text;
+  tags_list text;
+  tags_query text;
+  rec RECORD;
+BEGIN
+    SELECT translate($6::json->>'tag_name','[]', '{}')::text[] INTO tag_name;
+    SELECT array_to_string(tag_name, ',') INTO tags_list;
+    tags_query := '';
+
+    FOREACH tag IN ARRAY tag_name
+    LOOP
+      SELECT tags_query || ' sum(' || tag || '/fraction)::double precision as ' || tag || ', ' INTO tags_query;
+
+    END LOOP;
+
+    -- Simple mock, there should be real logic in here.
+    data_query := '(WITH _areas AS(SELECT ST_Area(a.the_geom::geography)'
+        || '/ (1000 * 1000) as fraction, a.geoid, b.cartodb_id FROM '
+        || 'observatory.obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308 as a, '
+        || table_schema || '.' || table_name || ' AS b '
+        || 'WHERE b.the_geom && a.the_geom ), values AS (SELECT geoid, '
+        || tags_list
+        || ' FROM observatory.obs_1a098da56badf5f32e336002b0a81708c40d29cd ) '
+        || 'SELECT '
+        || tags_query
+        || ' cartodb_id::int FROM _areas, values '
+        || 'WHERE values.geoid = _areas.geoid GROUP BY cartodb_id);';
+
+
+    FOR rec IN EXECUTE data_query
+    LOOP
+        RETURN NEXT rec;
+    END LOOP;
+    RETURN;
+END;
+$$ LANGUAGE plpgsql SECURITY DEFINER;
+
+
+CREATE OR REPLACE FUNCTION cdb_observatory._OBS_DisconnectUserTable(username text, orgname text, table_schema text, table_name text, servername text)
+RETURNS boolean
+AS $$
+BEGIN
+    EXECUTE 'DROP FOREIGN TABLE IF EXISTS "' || table_schema || '".' || table_name;
+    EXECUTE 'DROP SCHEMA IF EXISTS ' || table_schema || ' CASCADE';
+    EXECUTE 'DROP SERVER IF EXISTS ' || servername || ' CASCADE;';
+    RETURN true;
+END;
+$$ LANGUAGE plpgsql SECURITY DEFINER;

src/pg/test/expected/40_observatory_utility_test.out

@@ -9,21 +9,15 @@ t
 _obs_geomtable_with_null_response
 t
 (1 row)
-test_get_obs_column_with_geoid_and_census_1|test_get_obs_column_with_geoid_and_census_2
-t|t
-(1 row)
-obs_getcolumndata_missing_measure
-t
-(1 row)
 _obs_buildsnapshotquery_test_1
 t
 (1 row)
 _obs_buildsnapshotquery_test_2
 t
 (1 row)
-_obs_getrelatedcolumn_test
-t
-(1 row)
 _obs_standardizemeasurename_test
 t
 (1 row)
+obs_dumpversion_notnull
+t
+(1 row)

src/pg/test/expected/41_observatory_augmentation_test.out

@@ -1,5 +1,4 @@
-\i test/fixtures/load_fixtures.sql
-SET client_min_messages TO WARNING;
+\pset format unaligned
 \set ECHO none
 obs_getdemographicsnapshot_test_no_returns
 t
@@ -76,3 +75,15 @@ t
 obs_getuscensuscategory_polygon
 t
 (1 row)
+obs_getmeasurebyid_cartodb_census_tract
+t
+(1 row)
+obs_getmeasurebyid_null_boundary_null_timespan
+t
+(1 row)
+obs_getmeasurebyid_cartodb_block_group
+t
+(1 row)
+obs_getmeasurebyid_nulls
+t
+(1 row)

src/pg/test/expected/42_observatory_exploration_test.out

@@ -1,5 +1,4 @@
-\i test/fixtures/load_fixtures.sql
-SET client_min_messages TO WARNING;
+\pset format unaligned
 \set ECHO none
 _obs_searchtables_tables_match|_obs_searchtables_timespan_matches
 t|t
@@ -7,64 +6,9 @@ t|t
 _obs_searchtables_timespan_does_not_match
 t
 (1 row)
-id|description|name|aggregate|source
-us.census.acs.B01003001_quantile|The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates.|Quantile:Total Population|quantile|
-es.ine.total_pop|The total number of all people living in a geographic area.|Total Population|sum|
-us.census.acs.B01003001|The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates.|Total Population|sum|
-us.census.acs.B19301001|Per capita income is the mean income computed for every man, woman, and child in a particular group. It is derived by dividing the total income of a particular group by the total population.|Per Capita Income in the past 12 Months|average|
-us.census.acs.B01001001_quantile|The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates.|Quantile:Total Population|quantile|
-(5 rows)
-boundary_id|description|time_span|tablename
-us.census.tiger.zcta5|ZCTAs are approximate area representations of U.S. Postal Service (USPS) five-digit ZIP Code service areas that the Census Bureau creates using whole blocks to present statistical data from censuses and surveys. The Census Bureau defines ZCTAs by allocating each block that contains addresses to a single ZCTA, usually to the ZCTA that reflects the most frequently occurring ZIP Code for the addresses within that tabulation block. Blocks that do not contain addresses but are completely surrounded by a single ZCTA (enclaves) are assigned to the surrounding ZCTA; those surrounded by multiple ZCTAs will be added to a single ZCTA based on limited buffering performed between multiple ZCTAs. The Census Bureau identifies five-digit ZCTAs using a five-character numeric code that represents the most frequently occurring USPS ZIP Code within that ZCTA, and this code may contain leading zeros.
-There are significant changes to the 2010 ZCTA delineation from that used in 2000. Coverage was extended to include the Island Areas for 2010 so that the United States, Puerto Rico, and the Island Areas have ZCTAs. Unlike 2000, when areas that could not be assigned to a ZCTA were given a generic code ending in “XX” (land area) or “HH” (water area), for 2010 there is no universal coverage by ZCTAs, and only legitimate five-digit areas are defined. The 2010 ZCTAs will better represent the actual Zip Code service areas because the Census Bureau initiated a process before creation of 2010 blocks to add block boundaries that split polygons with large numbers of addresses using different Zip Codes.
-Data users should not use ZCTAs to identify the official USPS ZIP Code for mail delivery. The USPS makes periodic changes to ZIP Codes to support more efficient mail delivery. The ZCTAs process used primarily residential addresses and was biased towards Zip Codes used for city-style mail delivery, thus there may be Zip Codes that are primarily nonresidential or boxes only that may not have a corresponding ZCTA.|2014|obs_144e8b4f906885b2e057ac4842644a553ae49c6e
-whosonfirst.wof_continent_geom|Continents of the world.|2016|obs_9880042f935aab0d0e4b71fb6963d7726e64c534
-whosonfirst.wof_country_geom| |2016|obs_1ea93bbc109c87c676b3270789dacf7a1430db6c
-whosonfirst.wof_region_geom| |2016|obs_4fca4f060854cc3ae8c109999635a71bbde6964e
-whosonfirst.wof_marinearea_geom| |2016|obs_5105018d57c69b8a7e064fc17a9485647b311a99
-whosonfirst.wof_disputed_geom| |2016|obs_7dba9374b15fbab0c7bd7dca6dec6c4792fe86a3
-us.census.tiger.congressional_district|Congressional districts are the 435 areas from which people are elected to the U.S. House of Representatives. After the apportionment of congressional seats among the states based on decennial census population counts, each state with multiple seats is responsible for establishing congressional districts for the purpose of electing representatives. Each congressional district is to be as equal in population to all other congressional districts in a state as practicable. For the District of Columbia, Puerto Rico, and each Island Area, a separate code is used to identify the entire areas of these state-equivalent entities as having a single nonvoting delegate.|2014|obs_01b608b3a4ca503ad7acc0b1f84817bc1da3e193
-us.census.tiger.congressional_district_clipped|A cartography-ready version of US Congressional Districts|2014|obs_9d258128ff4288eb9a6d7e5c0adbef8ef0172a86
-us.census.tiger.school_district_unified_clipped|A cartography-ready version of Unified School District|2014|obs_546285f2c636f5380e7bfbb0c0db67863f6ed41d
-us.census.tiger.cbsa_clipped|A cartography-ready version of Core Based Statistical Area (CBSA)|2014|obs_3512a78ca8c7e9b5fbd5390c4ed1638b9938fcbf
-us.census.tiger.block|Census blocks are statistical areas bounded by visible features, such as streets, roads, streams, and railroad tracks, and by nonvisible boundaries, such as selected property lines and city, township, school district, and county limits and short line-of-sight extensions of streets and roads. Generally, census blocks are small in area; for example, a block in a city bounded on all sides by streets. Census blocks in suburban and rural areas may be large, irregular, and bounded by a variety of features, such as roads, streams, and transmission lines. In remote areas, census blocks may encompass hundreds of square miles. Census blocks cover the entire territory of the United States, Puerto Rico, and the Island Areas. Census blocks nest within all other tabulated census geographic entities and are the basis for all tabulated data.|2014|obs_ffebc3eb689edab4faa757f75ca02c65d7db7327
-us.census.tiger.school_district_elementary_clipped|A cartography-ready version of Elementary School District|2014|obs_41d99a86857c05f63320ea44419a68831e74ac3d
-us.census.tiger.block_group_clipped|A cartography-ready version of US Census Block Groups|2014|obs_6c1309a64d8f3e6986061f4d1ca7b57743e75e74
-us.census.tiger.census_tract_clipped|A cartography-ready version of US Census Tracts|2014|obs_fcd4e4f5610f6764973ef8c0c215b2e80bec8963
-us.census.tiger.zcta5_clipped|A cartography-ready version of US Census Zip Code Tabulation Areas|2014|obs_7615e8622a68bfc5fe37c69c9880edfb40250103
-us.census.tiger.place|Incorporated places are those reported to the Census Bureau as legally in existence as of January 1, 2010, as reported in the latest Boundary and Annexation Survey (BAS), under the laws of their respective states. An incorporated place is established to provide governmental functions for a concentration of people as opposed to a minor civil division, which generally is created to provide services or administer an area without regard, necessarily, to population. Places always are within a single state or equivalent entity, but may extend across county and county subdivision boundaries. An incorporated place usually is a city, town, village, or borough, but can have other legal descriptions. For Census Bureau data tabulation and presentation purposes, incorporated places exclude:
-Boroughs in Alaska (treated as statistical equivalents of counties).
-Towns in the New England states, New York, and Wisconsin (treated as MCDs).
-Boroughs in New York (treated as MCDs).
-Census Designated Places (CDPs) are the statistical counterparts of incorporated places, and are delineated to provide data for settled concentrations of population that are identifiable by name but are not legally incorporated under the laws of the state in which they are located. The boundaries usually are defined in cooperation with local or tribal officials and generally updated prior to each decennial census. These boundaries, which usually coincide with visible features or the boundary of an adjacent incorporated place or another legal entity boundary, have no legal status, nor do these places have officials elected to serve traditional municipal functions. CDP boundaries may change from one decennial census
-to the next with changes in the settlement pattern; a CDP with the same name as in an earlier census does not necessarily have the same boundary. CDPs must be contained within a single state and may not extend into an incorporated place. There are no population size requirements for CDPs.
-Hawaii is the only state that has no incorporated places recognized by the Census Bureau. All places shown in decennial census data products for Hawaii are CDPs. By agreement with the state of Hawaii, the Census Bureau does not show data separately for the city of Honolulu, which is coextensive with Honolulu County. In Puerto Rico, which also does not have incorporated places, the Census Bureau recognizes only CDPs and refers to them as comunidades or zonas urbanas. Guam also has only CDPs.|2014|obs_76a52df2018de8d064f1a99f93544473927cb7ae
-us.census.tiger.place_clipped|A cartography-ready version of Incorporated Places|2014|obs_db91d46d317a4ffcf509efca8e5e3a42d29e0792
-us.census.tiger.school_district_secondary_clipped|A cartography-ready version of Secondary School District|2014|obs_63b15ba742ccea136ddb88427cc37fee218702d4
-us.census.tiger.cbsa|Core Based Statistical Areas (CBSAs) consist of the county or counties or equivalent entities associated with at least one core (urbanized area or urban cluster) of at least 10,000 population, plus adjacent counties having a high degree of social and economic integration with the core as measured through commuting ties with the counties associated with the core. The general concept of a CBSA is that of a
-core area containing a substantial population nucleus, together with adjacent communities having a high degree of economic and social integration with that core. The term “core based statistical area” became effective in 2003 and refers collectively to metropolitan statistical areas and micropolitan statistical areas. The U.S. Office of Management and Budget (OMB) defines CBSAs to provide a nationally consistent set of geographic entities for the United States and Puerto Rico for use in tabulating and presenting statistical data. Current CBSAs are based on application of the 2000 standards (published in the Federal Register of December 27, 2000) with Census 2000 data. The first set of areas defined based on the 2000 standards were announced on June 6, 2003; subsequent updates have been made to the universe of CBSAs and related statistical areas. No CBSAs are defined in the Island Areas. Statistical areas related to CBSAs include metropolitan divisions, combined statistical areas (CSAs), New England city and town areas (NECTAs), NECTA divisions, and combined NECTAs.|2014|obs_c75be9ef45e87c789c3607dd9aeef6094d5e5109
-us.census.tiger.puma_clipped|A cartography-ready version of US Census Public Use Microdata Areas|2014|obs_dc244bf520f62e4a09e290a02e55368fd0758f95
-us.census.tiger.county|The primary legal divisions of most states are termed counties. In Louisiana, these divisions are known as parishes. In Alaska, which has no counties, the equivalent entities are the organized boroughs, city and boroughs, municipalities, and census areas; the latter of which are delineated cooperatively for statistical purposes by the state of Alaska and the Census Bureau. In four states (Maryland, Missouri, Nevada, and Virginia), there are one or more incorporated places that are independent of any county organization and thus constitute primary divisions of their states. These incorporated places are known as independent cities and are treated as equivalent entities for purposes of data presentation. The District of Columbia and Guam have no primary divisions, and each area is considered an equivalent entity for purposes of data presentation. All of the counties in Connecticut and Rhode Island and nine counties in Massachusetts were dissolved as functioning governmental entities; however, the Census Bureau continues to present data for these historical entities in order to provide comparable geographic units at the county level of the geographic hierarchy for these states and represents them as nonfunctioning legal entities in data products. The Census Bureau treats the following entities as equivalents of counties for purposes of data presentation: municipios in Puerto Rico, districts and islands in American Samoa, municipalities in the Commonwealth of the Northern Mariana Islands, and islands in the U.S. Virgin Islands. Each county or statistically equivalent entity is assigned a three-character numeric Federal Information Processing Series (FIPS) code based on alphabetical sequence that is unique within state and an eight-digit National Standard feature identifier.|2014|obs_1babf5a26a1ecda5fb74963e88408f71d0364b81
-us.census.tiger.county_clipped|A cartography-ready version of US County|2014|obs_23cb5063486bd7cf36f17e89e5e65cd31b331f6e
-us.census.tiger.school_district_unified|School Districts are geographic entities within which state, county, local officials, the Bureau of Indian Affairs, or the U.S. Department of Defense provide public educational services for the area’s residents. The Census Bureau obtains the boundaries, names, local education agency codes, and school district levels for school districts from state and local school officials for the primary purpose of providing the U.S. Department of Education with estimates of the number of children “at risk” within each school district, county, and state. This information serves as the basis for the Department of Education to determine the annual allocation of Title I funding to states and school districts.
-The Census Bureau tabulates data for three types of school districts: elementary, secondary, and unified. Each school district is assigned a five-digit code that is unique within state. School district codes are the local education agency number assigned by the Department of Education and are not necessarily in alphabetical order by school district name.
-The elementary school districts provide education to the lower grade/age levels and the secondary school districts provide education to the upper grade/age levels. Unified school districts provide education to children of all school ages in their service areas. In general, where there is a unified school district, no elementary or secondary school district exists; and where there is an elementary school district, the secondary school district may or may not exist.
-The Census Bureau’s representation of school districts in various data products is based both on the grade range that a school district operates and also the grade range for which the school district is financially responsible. For example, a school district is defined as an elementary school district if its operational grade range is less than the full kindergarten through 12 or prekindergarten through 12 grade range (for example, K–6 or pre-K–8). These elementary school districts do not provide direct educational services for grades 7–12, 9–12, or similar ranges. Some elementary school districts are financially responsible for the education of all school-aged children within their service areas and rely on other school districts to provide service for those grade ranges that are not operated by these elementary school districts. In these situations, in order to allocate all school-aged children to these school districts, the secondary school district code field is blank. For elementary school districts where the operational grade range and financially responsible grade range are the same, the secondary school district code field will contain a secondary school district code. There are no situations where an elementary school district does not exist and a secondary school district exists in Census Bureau records.|2014|obs_c948b5b1e4fdaf3302d888d0dc4cc821d8857c96
-us.census.tiger.census_tract|Census tracts are small, relatively permanent statistical subdivisions of a county or equivalent entity that are updated by local participants prior to each decennial census as part of the Census Bureau’s Participant Statistical Areas Program. The Census Bureau delineates census tracts in situations where no local participant existed or where state, local, or tribal governments declined to participate. The primary purpose of census tracts is to provide a stable set of geographic units for the presentation of statistical data.
-Census tracts generally have a population size between 1,200 and 8,000 people, with an optimum size of 4,000 people. A census tract usually covers a contiguous area; however, the spatial size of census tracts varies widely depending on the density of settlement. Census tract boundaries are delineated with the intention of being maintained over a long time so that statistical comparisons can be made from census to census. Census tracts occasionally are split due to population growth or merged as a result of substantial population decline.
-Census tract boundaries generally follow visible and identifiable features. They may follow nonvisible legal boundaries, such as minor civil division (MCD) or incorporated place boundaries in some states and situations, to allow for census-tract-to-governmental-unit relationships where the governmental boundaries tend to remain unchanged between censuses. State and county boundaries always are census tract boundaries in the standard census geographic hierarchy. Tribal census tracts are a unique geographic entity defined within federally recognized American Indian reservations and off-reservation trust lands and can cross state and county boundaries. Tribal census tracts may be completely different from the census tracts and block groups defined by state and county (see “Tribal Census Tract”).|2014|obs_fc050f0b8673cfe3c6aa1040f749eb40975691b7
-us.census.tiger.school_district_elementary|School Districts are geographic entities within which state, county, local officials, the Bureau of Indian Affairs, or the U.S. Department of Defense provide public educational services for the area’s residents. The Census Bureau obtains the boundaries, names, local education agency codes, and school district levels for school districts from state and local school officials for the primary purpose of providing the U.S. Department of Education with estimates of the number of children “at risk” within each school district, county, and state. This information serves as the basis for the Department of Education to determine the annual allocation of Title I funding to states and school districts.
-The Census Bureau tabulates data for three types of school districts: elementary, secondary, and unified. Each school district is assigned a five-digit code that is unique within state. School district codes are the local education agency number assigned by the Department of Education and are not necessarily in alphabetical order by school district name.
-The elementary school districts provide education to the lower grade/age levels and the secondary school districts provide education to the upper grade/age levels. Unified school districts provide education to children of all school ages in their service areas. In general, where there is a unified school district, no elementary or secondary school district exists; and where there is an elementary school district, the secondary school district may or may not exist.
-The Census Bureau’s representation of school districts in various data products is based both on the grade range that a school district operates and also the grade range for which the school district is financially responsible. For example, a school district is defined as an elementary school district if its operational grade range is less than the full kindergarten through 12 or prekindergarten through 12 grade range (for example, K–6 or pre-K–8). These elementary school districts do not provide direct educational services for grades 7–12, 9–12, or similar ranges. Some elementary school districts are financially responsible for the education of all school-aged children within their service areas and rely on other school districts to provide service for those grade ranges that are not operated by these elementary school districts. In these situations, in order to allocate all school-aged children to these school districts, the secondary school district code field is blank. For elementary school districts where the operational grade range and financially responsible grade range are the same, the secondary school district code field will contain a secondary school district code. There are no situations where an elementary school district does not exist and a secondary school district exists in Census Bureau records.|2014|obs_62ba00831d7f65309d6d663e2a99a688d336539a
-us.census.tiger.school_district_secondary|School Districts are geographic entities within which state, county, local officials, the Bureau of Indian Affairs, or the U.S. Department of Defense provide public educational services for the area’s residents. The Census Bureau obtains the boundaries, names, local education agency codes, and school district levels for school districts from state and local school officials for the primary purpose of providing the U.S. Department of Education with estimates of the number of children “at risk” within each school district, county, and state. This information serves as the basis for the Department of Education to determine the annual allocation of Title I funding to states and school districts.
-The Census Bureau tabulates data for three types of school districts: elementary, secondary, and unified. Each school district is assigned a five-digit code that is unique within state. School district codes are the local education agency number assigned by the Department of Education and are not necessarily in alphabetical order by school district name.
-The elementary school districts provide education to the lower grade/age levels and the secondary school districts provide education to the upper grade/age levels. Unified school districts provide education to children of all school ages in their service areas. In general, where there is a unified school district, no elementary or secondary school district exists; and where there is an elementary school district, the secondary school district may or may not exist.
-The Census Bureau’s representation of school districts in various data products is based both on the grade range that a school district operates and also the grade range for which the school district is financially responsible. For example, a school district is defined as an elementary school district if its operational grade range is less than the full kindergarten through 12 or prekindergarten through 12 grade range (for example, K–6 or pre-K–8). These elementary school districts do not provide direct educational services for grades 7–12, 9–12, or similar ranges. Some elementary school districts are financially responsible for the education of all school-aged children within their service areas and rely on other school districts to provide service for those grade ranges that are not operated by these elementary school districts. In these situations, in order to allocate all school-aged children to these school districts, the secondary school district code field is blank. For elementary school districts where the operational grade range and financially responsible grade range are the same, the secondary school district code field will contain a secondary school district code. There are no situations where an elementary school district does not exist and a secondary school district exists in Census Bureau records.|2014|obs_7b120df094ad943984b820d83ced7392af3a551a
-us.census.tiger.block_group|Block groups (BGs) are statistical divisions of census tracts, are generally defined to contain between 600 and 3,000 people, and are used to present data and control block numbering. A block group consists of clusters of blocks within the same census tract that have the same first digit of their four-digit census block number. For example, blocks 3001, 3002, 3003, ..., 3999 in census tract 1210.02 belong to BG 3 in that census tract. Most BGs were delineated by local participants in the Census Bureau’s Participant Statistical Areas Program. The Census Bureau delineated BGs only where a local or tribal government declined to participate, and a regional organization or State Data Center was not available to participate.
-A BG usually covers a contiguous area. Each census tract contains at least one BG, and BGs are uniquely numbered within the census tract. Within the standard census geographic hierarchy, BGs never cross state, county, or census tract boundaries but may cross the boundaries of any other geographic entity. Tribal census tracts and tribal BGs are separate and unique geographic areas defined within federally recognized American Indian reservations and can cross state and county boundaries (see “Tribal Census Tract” and “Tribal Block Group”). The tribal census tracts and tribal block groups may be completely different from the census tracts and block groups defined by state and county.|2014|obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308
-us.census.tiger.state|States and Equivalent Entities are the primary governmental divisions of the United States. In addition to the 50 states, the Census Bureau treats the District of Columbia, Puerto Rico, American Samoa, the Commonwealth of the Northern Mariana Islands, Guam, and the U.S. Virgin Islands as the statistical equivalents of states for the purpose of data presentation.|2014|obs_624e5d2362e08aaa5463d7671e7748432262719c
-us.census.tiger.state_clipped|A cartography-ready version of US States|2014|obs_f39f1d7cd5a22b87140860cbd58539f1591a1810
-us.census.tiger.puma|PUMAs are geographic areas for which the Census Bureau provides selected extracts of raw data from a small sample of census records that are screened to protect confidentiality. These extracts are referred to as public use microdata sample (PUMS) files.
-For the 2010 Census, each state, the District of Columbia, Puerto Rico, and some Island Area participants delineated PUMAs for use in presenting PUMS data based on a 5 percent sample of decennial census or American Community Survey data. These areas are required to contain at least 100,000 people. This is different from Census 2000 when two types of PUMAs were defined: a 5 percent PUMA as for 2010 and an additional super-PUMA designed to provide a 1 percent sample. The PUMAs are identified by a five-digit census code unique within state.|2014|obs_7c9493c41fa8f4bd178ab993ea3d5891c1977667
-(30 rows)
+_obs_searchtotalpop
+t
+(1 row)
+_obs_getavailableboundariesexist
+t
+(1 row)

src/pg/test/expected/44_observatory_geometries_test.out

@@ -1,7 +1,4 @@
 \pset format unaligned
-\set ECHO all
-\i test/fixtures/load_fixtures.sql
-SET client_min_messages TO WARNING;
 \set ECHO none
 obs_getboundary_cartodb_census_tract
 t
@@ -57,6 +54,9 @@ t
 obs_getboundariesbygeometry_tracts_around_null_island
 t
 (1 row)
+obs_getboundariesbygeometry_wof
+t
+(1 row)
 obs_getboundariesbypointandradius_around_cartodb
 t
 (1 row)
@@ -90,3 +90,6 @@ t
 geoid_name_matches|table_name_matches|geom_name_matches
 t|t|t
 (1 row)
+geoid_name_matches|table_name_matches|geom_name_matches
+t|t|t
+(1 row)

src/pg/test/fixtures/drop_fixtures.sql

@@ -6,6 +6,7 @@ DROP TABLE IF EXISTS observatory.obs_column;
 DROP TABLE IF EXISTS observatory.obs_column_tag;
 DROP TABLE IF EXISTS observatory.obs_tag;
 DROP TABLE IF EXISTS observatory.obs_column_to_column;
+DROP TABLE IF EXISTS observatory.obs_dump_version;
 DROP TABLE IF EXISTS observatory.obs_65f29658e096ca1485bf683f65fdbc9f05ec3c5d;
 DROP TABLE IF EXISTS observatory.obs_1746e37b7cd28cb131971ea4187d42d71f09c5f3;
 DROP TABLE IF EXISTS observatory.obs_1a098da56badf5f32e336002b0a81708c40d29cd;
@@ -13,8 +14,11 @@ DROP TABLE IF EXISTS observatory.obs_7615e8622a68bfc5fe37c69c9880edfb40250103;
 DROP TABLE IF EXISTS observatory.obs_1babf5a26a1ecda5fb74963e88408f71d0364b81;
 DROP TABLE IF EXISTS observatory.obs_8764a6b439a4f8714f54d4b3a157bc5e36519066;
 DROP TABLE IF EXISTS observatory.obs_b393b5b88c6adda634b2071a8005b03c551b609a;
+DROP TABLE IF EXISTS observatory.obs_1ea93bbc109c87c676b3270789dacf7a1430db6c;
 DROP TABLE IF EXISTS observatory.obs_fc050f0b8673cfe3c6aa1040f749eb40975691b7;
 DROP TABLE IF EXISTS observatory.obs_6c1309a64d8f3e6986061f4d1ca7b57743e75e74;
 DROP TABLE IF EXISTS observatory.obs_d39f7fe5959891c8296490d83c22ded31c54af13;
 DROP TABLE IF EXISTS observatory.obs_144e8b4f906885b2e057ac4842644a553ae49c6e;
 DROP TABLE IF EXISTS observatory.obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308;
+
+DROP TABLE IF EXISTS observatory.obs_meta;

src/pg/test/sql/40_observatory_utility_test.sql

@@ -1,6 +1,8 @@
 \pset format unaligned
 \set ECHO all
 \i test/fixtures/load_fixtures.sql
+SET client_min_messages TO WARNING;
+\set ECHO none
 
 -- OBS_GeomTable
 -- get table with known geometry_id
@@ -27,29 +29,6 @@ SELECT
 --     'us.census.tiger.census_tract'
 --   );
 
-WITH result as (
-SELECT
-  array_agg(a) expected from cdb_observatory._OBS_GetColumnData(
-    'us.census.tiger.census_tract',
-    Array['us.census.spielman_singleton_segments.X55', 'us.census.acs.B01003001'],
-         '2010 - 2014') a
-)
-select
-(expected)[1]::text  = '{"colname":"x55","tablename":"obs_65f29658e096ca1485bf683f65fdbc9f05ec3c5d","aggregate":null,"name":"Spielman-Singleton Segments: 55 Clusters","type":"Text","description":"Sociodemographic classes from Spielman and Singleton 2015, 55 clusters","boundary_id":"us.census.tiger.census_tract"}' as test_get_obs_column_with_geoid_and_census_1,
-(expected)[2]::text  = '{"colname":"total_pop","tablename":"obs_b393b5b88c6adda634b2071a8005b03c551b609a","aggregate":"sum","name":"Total Population","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates.","boundary_id":"us.census.tiger.census_tract"}' as test_get_obs_column_with_geoid_and_census_2
-from result;
-
--- should be null-valued
-WITH result as (
-SELECT
-  array_agg(a) expected from cdb_observatory._OBS_GetColumnData(
-    'us.census.tiger.census_tract',
-    Array['us.census.tiger.baloney'],
-    '2010 - 2014') a
-)
-select expected is null as OBS_GetColumnData_missing_measure
-from result;
-
 -- OBS_BuildSnapshotQuery
 -- Should give back: SELECT  vals[1] As total_pop, vals[2] As male_pop, vals[3] As female_pop, vals[4] As median_age
 SELECT
@@ -63,16 +42,8 @@ SELECT
     Array['mandarin_orange']
   ) = 'SELECT  vals[1] As mandarin_orange' As _OBS_BuildSnapshotQuery_test_2;
 
-SELECT cdb_observatory._OBS_GetRelatedColumn(
-    Array[
-     'es.ine.pop_0_4',
-     'us.census.acs.B01003001',
-     'us.census.acs.B01001002'
-    ],
-     'denominator'
- ) = '{es.ine.total_pop,NULL,us.census.acs.B01003001}' As _OBS_GetRelatedColumn_test;
-
 -- should give back a standardized measure name
 SELECT cdb_observatory._OBS_StandardizeMeasureName('test 343 %% 2 qqq }}{{}}') = 'test_343_2_qqq' As _OBS_StandardizeMeasureName_test;
 
-\i test/fixtures/drop_fixtures.sql
+SELECT cdb_observatory.OBS_DumpVersion()
+  IS NOT NULL AS OBS_DumpVersion_notnull;

src/pg/test/sql/41_observatory_augmentation_test.sql

@@ -1,6 +1,6 @@
-\i test/fixtures/load_fixtures.sql
 \pset format unaligned
 \set ECHO none
+SET client_min_messages TO WARNING;
 
 --
 WITH result as(
@@ -131,7 +131,7 @@ WITH result as (
 -- Point-based OBS_GetMeasure with zillow
 SELECT abs(OBS_GetMeasure_zhvi_point - 583600) / 583600 < 0.001 AS OBS_GetMeasure_zhvi_point_test FROM cdb_observatory.OBS_GetMeasure(
   ST_SetSRID(ST_Point(-73.94602417945862, 40.6768220087458), 4326),
-  'us.zillow.AllHomes_Zhvi', 'area', 'us.census.tiger.zcta5', '2014-01'
+  'us.zillow.AllHomes_Zhvi', null, 'us.census.tiger.zcta5', '2014-01'
 ) As t(OBS_GetMeasure_zhvi_point);
 
 -- Point-based OBS_GetMeasure with zillow default to latest
@@ -150,7 +150,7 @@ SELECT abs(OBS_GetMeasure_total_pop_point - 10923.093200390833950) / 10923.09320
 
 -- Poly-based OBS_GetMeasure, default normalization (none)
 -- is result within 0.1% of expected
-SELECT abs(OBS_GetMeasure_total_pop_polygon - 9833.47316573952) / 9833.47316573952 < 0.001 As OBS_GetMeasure_total_pop_polygon_test FROM
+SELECT abs(OBS_GetMeasure_total_pop_polygon - 12327.3133495107) / 12327.3133495107 < 0.001 As OBS_GetMeasure_total_pop_polygon_test FROM
   cdb_observatory.OBS_GetMeasure(
     cdb_observatory._TestArea(),
     'us.census.acs.B01003001'
@@ -164,7 +164,7 @@ SELECT (abs(cdb_observatory.OBS_GetMeasure(
 -- Poly-based OBS_GetMeasure with denominator normalization
 SELECT abs(cdb_observatory.OBS_GetMeasure(
   cdb_observatory._TestArea(),
-  'us.census.acs.B01001002', 'denominator') - 0.50597531462834994530) / 0.49026340444793965457 < 0.001 As OBS_GetMeasure_total_male_poly_denominator;
+  'us.census.acs.B01001002', 'denominator') - 0.49026340444793965457) / 0.49026340444793965457 < 0.001 As OBS_GetMeasure_total_male_poly_denominator;
 
 -- Point-based OBS_GetCategory
 SELECT cdb_observatory.OBS_GetCategory(
@@ -172,7 +172,7 @@ SELECT cdb_observatory.OBS_GetCategory(
 
 -- Poly-based OBS_GetCategory
 SELECT cdb_observatory.OBS_GetCategory(
-  cdb_observatory._TestArea(), 'us.census.spielman_singleton_segments.X10') = 'Low income, mix of minorities' As obs_getcategory_polygon;
+  cdb_observatory._TestArea(), 'us.census.spielman_singleton_segments.X10') = 'Wealthy, urban without Kids' As obs_getcategory_polygon;
 
 -- Point-based OBS_GetPopulation, default normalization (area)
 SELECT (abs(OBS_GetPopulation - 10923.093200390833950) / 10923.093200390833950) < 0.001 As OBS_GetPopulation FROM
@@ -181,7 +181,7 @@ SELECT (abs(OBS_GetPopulation - 10923.093200390833950) / 10923.093200390833950)
   ) As m(OBS_GetPopulation);
 
 -- Poly-based OBS_GetPopulation, default normalization (none)
-SELECT (abs(obs_getpopulation_polygon - 9833.47316573952) / 9833.47316573952) < 0.001 As obs_getpopulation_polygon_test
+SELECT (abs(obs_getpopulation_polygon - 12327.3133495107) / 12327.3133495107) < 0.001 As obs_getpopulation_polygon_test
 FROM
   cdb_observatory.OBS_GetPopulation(
     cdb_observatory._TestArea()
@@ -193,7 +193,7 @@ SELECT (abs(cdb_observatory.obs_getuscensusmeasure(
 
 -- Poly-based OBS_GetUSCensusMeasure, default normalization (none)
 SELECT (abs(cdb_observatory.obs_getuscensusmeasure(
-  cdb_observatory._testarea(), 'male population') - 4975.49467892449) / 4975.49467892449) < 0.001 As obs_getuscensusmeasure;
+  cdb_observatory._testarea(), 'male population') - 6043.63061042765) / 6043.63061042765) < 0.001 As obs_getuscensusmeasure;
 
 -- Point-based OBS_GetUSCensusCategory
 SELECT cdb_observatory.OBS_GetUSCensusCategory(
@@ -201,7 +201,38 @@ SELECT cdb_observatory.OBS_GetUSCensusCategory(
 
 -- Area-based OBS_GetUSCensusCategory
 SELECT cdb_observatory.OBS_GetUSCensusCategory(
-  cdb_observatory._testarea(), 'Spielman-Singleton Segments: 10 Clusters') = 'Low income, mix of minorities' As OBS_GetUSCensusCategory_polygon;
+  cdb_observatory._testarea(), 'Spielman-Singleton Segments: 10 Clusters') = 'Wealthy, urban without Kids' As OBS_GetUSCensusCategory_polygon;
 
 
-\i test/fixtures/drop_fixtures.sql
+-- OBS_GetMeasureById tests
+-- typical query
+SELECT (cdb_observatory.OBS_GetMeasureById(
+  '36047048500',
+  'us.census.acs.B01003001',
+  'us.census.tiger.census_tract',
+  '2010 - 2014'
+) - 3241) / 3241 < 0.0001 As OBS_GetMeasureById_cartodb_census_tract;
+
+-- no boundary_id should give null
+SELECT cdb_observatory.OBS_GetMeasureById(
+  '36047048500',
+  'us.census.acs.B01003001',
+  NULL,
+  NULL
+) IS NULL As OBS_GetMeasureById_null_boundary_null_timespan;
+
+-- query at block_group level
+SELECT (cdb_observatory.OBS_GetMeasureById(
+  '360470485002',
+  'us.census.acs.B01003001',
+  'us.census.tiger.block_group',
+  '2010 - 2014'
+) - 1900) / 1900 < 0.0001 As OBS_GetMeasureById_cartodb_block_group;
+
+-- geom ref / boundary mismatch
+SELECT cdb_observatory.OBS_GetMeasureById(
+  '36047048500',
+  'us.census.acs.B01003001',
+  'us.census.tiger.block_group',
+  '2010 - 2014'
+) IS NULL As OBS_GetMeasureById_nulls;

src/pg/test/sql/42_observatory_exploration_test.sql

@@ -1,5 +1,6 @@
-\i test/fixtures/load_fixtures.sql
 \pset format unaligned
+\set ECHO none
+SET client_min_messages TO WARNING;
 
 -- set up variables for use in testing
 
@@ -24,10 +25,11 @@ FROM cdb_observatory._OBS_SearchTables(
   '1988' -- year before first tiger data was collected
 ) As t(table_name, timespan);
 
-SELECT *
-FROM cdb_observatory.OBS_Search('total_pop');
+SELECT COUNT(*) > 0 AS _OBS_SearchTotalPop
+FROM cdb_observatory.OBS_Search('total_pop')
+AS t(id, description, name, aggregate, source);
 
-SELECT *
-FROM cdb_observatory.OBS_GetAvailableBoundaries(cdb_observatory._TestPoint());
-
-\i test/fixtures/drop_fixtures.sql
+SELECT COUNT(*) > 0 AS _OBS_GetAvailableBoundariesExist
+FROM cdb_observatory.OBS_GetAvailableBoundaries(
+  cdb_observatory._TestPoint()
+) AS t(boundary_id, description, time_span, tablename);

src/pg/test/sql/44_observatory_geometries_test.sql

@@ -1,6 +1,6 @@
 \pset format unaligned
-\set ECHO all
-\i test/fixtures/load_fixtures.sql
+\set ECHO none
+SET client_min_messages TO WARNING;
 
 -- set up variables for use in testing
 
@@ -171,6 +171,17 @@ FROM (
   ORDER BY geom_refs ASC
 ) As m(the_geom, geom_refs);
 
+-- who's on first boundaries
+SELECT
+  array_agg(geom_refs) = Array['85632785','85633051','85633111','85633147','85633253','85633267'] As OBS_GetBoundariesByGeometry_wof
+FROM (
+  SELECT *
+  FROM cdb_observatory.OBS_GetBoundariesByGeometry(
+    ST_MakeEnvelope(-4.66, 40.43, 14.48, 51.99, 4326),
+    'whosonfirst.wof_country_geom')
+  ORDER BY geom_refs ASC
+) As m(the_geom, geom_refs);
+
 -- OBS_GetBoundariesByPointAndRadius
 
 -- check that all census tracts intersecting with the geometry are returned
@@ -320,6 +331,7 @@ FROM (
 ) As m(the_geom, geom_refs);
 
 -- _OBS_GetGeometryMetadata
+-- get metadata for census tracts
 
 SELECT
   geoid_colname = 'geoid' As geoid_name_matches,
@@ -328,4 +340,11 @@ SELECT
 FROM cdb_observatory._OBS_GetGeometryMetadata('us.census.tiger.census_tract')
      As m(geoid_colname, target_table, geom_colname);
 
+-- get metadata for boundaries with clipped geometries
+ SELECT
+   geoid_colname = 'geoid' As geoid_name_matches,
+   target_table = 'obs_fcd4e4f5610f6764973ef8c0c215b2e80bec8963' As table_name_matches,
+   geom_colname = 'the_geom' As geom_name_matches
+ FROM cdb_observatory._OBS_GetGeometryMetadata('us.census.tiger.census_tract_clipped') As m(geoid_colname, target_table, geom_colname);
+
 \i test/fixtures/drop_fixtures.sql