release v1.3.1

Release 1.3.0

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,23 @@
+## Request for a new Data observatory extension deploy
+
+I'd like to request a new data observatory extension deploy: dump + extension
+
+## Performance comparison to last deployment
+
+Please include link here to comparison perftests:
+
+http://52.71.151.140/perftest/#oldsha..newsha
+
+## Dump database id to be deployed
+
+Please put here the dump id to be deployed: <dump_id>
+
+## Data Observatory extension PRs included.
+
+*Please update the NEWS.md*
+
+Add down here the PR links to be added and deployed:
+
+  - 
+
+// @CartoDB/dataservices

Makefile

@@ -18,7 +18,7 @@ test:   ## Run the tests for the development version of the extension
 	$(MAKE) -C $(EXT_DIR) test
 
 # Generate a new release into release
-release: ## Generate a new release of the extension. Only for telease manager
+release: ## Generate a new release of the extension. Only for release manager
 	$(MAKE) -C $(EXT_DIR) release
 
 # Install the current release.

NEWS.md

@@ -1,3 +1,199 @@
+1.3.1 (2017-02-16)
+
+__Improvements__
+
+* It is now possible to obtain measures that are averages or medians over
+  arbitrary polygons ([#254](https://github.com/CartoDB/observatory-extension/pull/254).
+* Added test point for Australian data
+* `OBS_GetLegacyMetadata` now returns median and averages in cases where it is
+  called for measures for polygons
+
+1.3.0 (2017-01-17)
+
+__API Changes__
+
+* `OBS_GetMeasureDataMulti()` is now called `OBS_GetData()`
+* `OBS_GetMeasureMetaMulti()` is now called `OBS_GetMeta()`
+* Additional signature for `OBS_GetData` which can take an array of `TEXT`,
+  mimicking functionality of `OBS_GetMeasureByID`
+
+__Improvements__
+
+* Generate fixtures from `obs_meta`
+* Remove unused table-level code
+* Refactor all augmentation and geometry functions to obtain data from
+  `OBS_GetMeta()` and `OBS_GetData()`.
+* Improvements to `OBS_GetMeta()` so it can still fill in metadata in cases
+  where only a geometry is being requested.
+* `OBS_GetData()` returns two-column table instead of anonymous record.
+* `OBS_GetData()` can return categorical (text) and geometries
+
+__Bugfixes__
+
+* Remove unnecessary dependency on `postgres_fdw`
+* `OBS_GetData()` now aggregates measures with mixed geoms correctly
+
+__API Changes__
+
+1.2.1 (2017-01-17)
+
+__Improvements__
+
+* Support Point/LineString in responses from `OBS_GetBoundary`.
+  ([#243](https://github.com/CartoDB/observatory-extension/pull/233))
+
+1.2.0 (2016-12-28)
+
+__API Changes__
+
+* Added `OBS_GetMeasureDataMulti`, which takes an array of geomvals and
+  parameters as JSON, and returns a set of RECORDs keyed by the vals of the
+  geomvals.
+* Added `OBS_GetMeasureMetaMulti`, which takes sparse metadata as JSON (for
+  example, the measure ID) and returns a filled-out version of the metadata
+  sufficient for use with `OBS_GetMeasureDataMulti`.
+
+__Improvements__
+
+* Move tests to 2015
+* Fixes to `_OBS_GetGeometryScores` to avoid spamming NOTICEs about all pixels
+  for a band being NULL
+* Tests for `_OBS_GetGeometryScores` with complex geometries
+* Performance tests for `OBS_GetMeasureDataMulti`
+* Return both `table_id` and `column_id` from `_OBS_GetGeometryScores`
+
+1.1.7 (2016-12-15)
+
+__Improvements__
+
+* Use simpler raster table and simplified `_OBSGetGeometryScores` functions to
+  improve performance
+* In cases where geometry passed into geometry scoring function has greater
+  than 10K points, simply use its buffer instead
+* Add `IMMUTABLE` to `_OBSGetGeometryScores`
+* Add tests explicitly for `_OBSGetGeometryScores` in perftest.py
+* Yields a ~50% improvement in performance for `_OBSGetGeomeryScores`.
+
+1.1.6 (2016-12-08)
+
+__Bugfixes__
+
+* Fix divide by zero condition in "denominator" branch of `OBS_GetMeasure`
+  when passing in a polygon ([#233](https://github.com/CartoDB/observatory-extension/pull/233)).
+
+__Improvements__
+
+* Use `ST_Subdivide` to improve performance when functions are called on very
+  complex geometries (with many points) ([#232](https://github.com/CartoDB/observatory-extension/pull/232))
+* Improve raster scoring to more heavily weight boundaries with nearer to
+  correct number of points, and penalize boundaries with lots of blank space
+  ([#232](https://github.com/CartoDB/observatory-extension/pull/232))
+* Remove some redundant area calculations in `OBS_GetMeasure`
+  ([#232](https://github.com/CartoDB/observatory-extension/pull/232))
+* Replace use of `format('%L', var)` with proper use of `EXECUTE` and `$1` etc.
+  variables ([#231](https://github.com/CartoDB/observatory-extension/pull/231))
+* Add test point for Brazil
+  ([#229](https://github.com/CartoDB/observatory-extension/pull/229))
+* Improvements to performance tests
+  ([#229](https://github.com/CartoDB/observatory-extension/pull/229))
+  - Support simple and complex geometries
+  - Handle all code branches
+  - Add ability to persist results to JSON for graph visualization later
+
+1.1.5 (2016-11-29)
+
+__Bugfixes__
+
+* Return `NULL` instead of raising an exception when a measure is requested for
+  a geometry where it does not exist ([#220](https://github.com/CartoDB/observatory-extension/issues/220)).
+
+1.1.4 (2016-11-21)
+
+__Bugfixes__
+
+* Fix duplicate subsections with only a partial set of measures appearing from
+  `OBS_GetLegacyMetadata` ([#216](https://github.com/CartoDB/observatory-extension/issues/216)).
+
+1.1.3 (2016-11-15)
+
+* Temporarily ignore EU data for the sake of testing
+
+1.1.2 (2016-11-09)
+
+__Improvements__
+
+* Update public `OBS_GetMeasure` to use highest ranked boundary, aiming for 500
+  geoms. ([#190](https://github.com/CartoDB/observatory-extension/issues/190))
+* Update test generation to capture our raster tiles
+* Standardize the way we generate our test points for `autotest.py`
+* Add points for epa and eurostat
+* Should support database dump generated 20161109
+
+__API Changes (Internal)__
+
+* Add internal `_OBS_GetGeometryScores`
+
+1.1.1 (2016-10-14)
+
+__Improvements__
+
+* Test points for Canada and France ([#204](https://github.com/CartoDB/observatory-extension/issues/120))
+
+1.1.0 (2016-10-04)
+
+__Bugfixes__
+
+* Fixed some minor errors in test suite
+
+__Improvements__
+
+* We now generate test fixtures from local data instead of remote server
+  ([#120](https://github.com/CartoDB/observatory-extension/issues/120))
+
+__API Changes__
+
+* New function, `OBS_LegacyBuilderMetadata`, which resolves
+  ([#133]( https://github.com/CartoDB/observatory-extension/issues/133))
+* Creates "dimensional" metadata grabbing functions
+  (`OBS_GetAvailableNumerators`, `OBS_GetAvailableDenominators`,
+  `OBS_GetAvailableGeometries`, `OBS_GetAvailableTimespans`) which will be
+  used for obtaining metadata in the replacement for the Data Library
+  ([CartoDB/design#104](https://github.com/CartoDB/design/issues/104)). This
+  is also referred to here ([CartoDB/design#68](https://github.com/CartoDB/design/issues/68)).
+
+1.0.7 (2016-09-20)
+
+__Bugfixes__
+
+* `NULL` geometries or geometry IDs no longer result in an exception from any
+  augmentation functions ([#178](https://github.com/CartoDB/observatory-extension/issues/178))
+
+__Improvements__
+
+* Automatic tests work for Canada and Thailand
+
+1.0.6 (2016-09-08)
+
+__Improvements__
+
+* New function structure for Table-level functions which allows to separate the
+  framework logic from the observatory measure functions.
+
+1.0.5 (2016-08-12)
+
+__Improvements__
+
+* Integration tests moved to `src/python/test/`, and can be run without hitting
+  any HTTP SQL API.
+
+1.0.4 (2016-07-26)
+
+__Bugfixes__
+
+* Always default arguments to `NULL`, which prevents duplication & overwrite by
+  dataservices-api
+  ([#173](https://github.com/CartoDB/observatory-extension/issues/173))
+
 1.0.3 (2016-07-25)
 
 __Bugfixes__

doc/boundary_functions.md

@@ -2,7 +2,7 @@
 
 Use the following functions to retrieve [Boundary](https://carto.com/docs/carto-engine/data/overview/#boundary-data) data. Data ranges from small areas (e.g. US Census Block Groups) to large areas (e.g. Countries). You can access boundaries by point location lookup, bounding box lookup, direct ID access and several other methods described below.
 
-You can [access](https://carto.com/docs/carto-engine/data/accessing) boundaries through the CARTO Editor. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](http://docs/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
+You can [access](https://carto.com/docs/carto-engine/data/accessing) boundaries through CARTO Builder. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](http://docs/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
 
 ## OBS_GetBoundariesByGeometry(polygon geometry, geometry_id text)
 

doc/measures_functions.md

@@ -2,9 +2,9 @@
 
 [Data Observatory Measures](https://carto.com/docs/carto-engine/data/overview/#measures-methods) are the numerical location data you can access. The measure functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (point or polygon) and receiving back a measure (an absolute value) for that location.
 
-There are hundreds of measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf).
+There are hundreds of measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](https://cartodb.github.io/bigmetadata/index.html).
 
-You can [access](https://carto.com/docs/carto-engine/data/accessing) measures through the CARTO Editor. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](https://carto.com/docs/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
+You can [access](https://carto.com/docs/carto-engine/data/accessing) measures through CARTO Builder. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](https://carto.com/docs/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
 
 ## OBS_GetUSCensusMeasure(point geometry, measure_name text)
 
@@ -16,7 +16,7 @@ Name |Description
 --- | ---
 point | a WGS84 point geometry (the_geom)
 measure_name | a human readable name of a US Census variable. The list of measure_names is [available in the Glossary](https://carto.com/docs/carto-engine/data/glossary/#obsgetuscensusmeasure-names-table).
-normalize | for measures that are are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf) (optional)
+normalize | for measures that are are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/index.html) (optional)
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span | time span of interest (e.g., 2010 - 2014)
 
@@ -47,7 +47,7 @@ Name |Description
 --- | ---
 polygon | a WGS84 polygon geometry (the_geom)
 measure_name | a human readable string name of a US Census variable. The list of measure_names is [available in the Glossary](https://carto.com/docs/carto-engine/data/glossary/#obsgetuscensusmeasure-names-table).
-normalize | for measures that are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf) (optional)
+normalize | for measures that are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/index.html) (optional)
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span | time span of interest (e.g., 2010 - 2014)
 
@@ -70,7 +70,7 @@ SET local_male_population = OBS_GetUSCensusMeasure(the_geom, 'Male Population')
 
 ## OBS_GetMeasure(point geometry, measure_id text)
 
-The ```OBS_GetMeasure(point, measure_id)``` function returns any Data Observatory measure at a point location. You can browse all available Measures in the [Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf).
+The ```OBS_GetMeasure(point, measure_id)``` function returns any Data Observatory measure at a point location. You can browse all available Measures in the [Catalog](https://cartodb.github.io/bigmetadata/index.html).
 
 #### Arguments
 
@@ -78,7 +78,7 @@ Name |Description
 --- | ---
 point | a WGS84 point geometry (the_geom)
 measure_id | a measure identifier from the Data Observatory ([see available measures](https://cartodb.github.io/bigmetadata/observatory.pdf)). It is important to note that these are different than 'measure_name' used in the Census based functions above.
-normalize | for measures that are are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. The other option is 'denominator', which will use the denominator specified in the [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf). (optional)
+normalize | for measures that are are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. The other option is 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/index.html). (optional)
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span | time span of interest (e.g., 2010 - 2014)
 
@@ -109,7 +109,7 @@ Name |Description
 --- | ---
 polygon_geometry | a WGS84 polygon geometry (the_geom)
 measure_id | a measure identifier from the Data Observatory ([see available measures](https://cartodb.github.io/bigmetadata/observatory.pdf))  
-normalize | for measures that are are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf) (optional)
+normalize | for measures that are are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/index.html) (optional)
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span | time span of interest (e.g., 2010 - 2014)
 
@@ -170,7 +170,7 @@ SET household_count = OBS_GetMeasureById(geoid_column, 'us.census.acs.B11001001'
 
 ## OBS_GetCategory(point geometry, category_id text)
 
-The ```OBS_GetCategory(point, category_id)``` function returns any Data Observatory Category value at a point location. The Categories available are currently limited to Segmentation categories. See the Segmentation section of the [Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf) for more detail.
+The ```OBS_GetCategory(point, category_id)``` function returns any Data Observatory Category value at a point location. The Categories available are currently limited to Segmentation categories. See the Segmentation section of the [Catalog](https://cartodb.github.io/bigmetadata/index.html) for more detail.
 
 #### Arguments
 

release/observatory.control

@@ -1,5 +1,5 @@
 comment = 'CartoDB Observatory backend extension'
-default_version = '1.0.3'
-requires = 'postgis, postgres_fdw'
+default_version = '1.3.1'
+requires = 'postgis'
 superuser = true
 schema = cdb_observatory

scripts/autotest.py

@@ -1,240 +0,0 @@
-from nose.tools import assert_equal, assert_is_not_none
-from nose_parameterized import parameterized
-
-import os
-import re
-import requests
-
-HOSTNAME = os.environ['OBS_HOSTNAME']
-API_KEY = os.environ['OBS_API_KEY']
-META_HOSTNAME = os.environ.get('OBS_META_HOSTNAME', HOSTNAME)
-META_API_KEY = os.environ.get('OBS_META_API_KEY', API_KEY)
-USE_SCHEMA = 'OBS_USE_SCHEMA' in os.environ
-
-
-def query(q, is_meta=False, **options):
-    '''
-    Query the account.  Returned is the response, wrapped by the requests
-    library.
-    '''
-    url = 'https://{hostname}/api/v2/sql'.format(
-        hostname=META_HOSTNAME if is_meta else HOSTNAME)
-    params = options.copy()
-    params['q'] = re.sub(r'\s+', ' ', q)
-    params['api_key'] = META_API_KEY if is_meta else API_KEY
-    return requests.get(url, params=params)
-
-MEASURE_COLUMNS = [(r['numer_id'], r['point_only'], ) for r in query('''
-SELECT distinct numer_id, numer_aggregate NOT ILIKE 'sum' as point_only
-FROM obs_meta
-WHERE numer_type ILIKE 'numeric'
-AND numer_weight > 0
-''', is_meta=True).json()['rows']]
-
-CATEGORY_COLUMNS = [(r['numer_id'], ) for r in query('''
-SELECT distinct numer_id
-FROM obs_meta
-WHERE numer_type ILIKE 'text'
-AND numer_weight > 0
-''', is_meta=True).json()['rows']]
-
-BOUNDARY_COLUMNS = [(r['id'], ) for r in query('''
-SELECT id FROM obs_column
-WHERE type ILIKE 'geometry'
-AND weight > 0
-''', is_meta=True).json()['rows']]
-
-US_CENSUS_MEASURE_COLUMNS = [(r['numer_name'], ) for r in query('''
-SELECT distinct numer_name
-FROM obs_meta
-WHERE numer_type ILIKE 'numeric'
-AND 'us.census.acs.acs' = ANY (subsection_tags)
-AND numer_weight > 0
-''', is_meta=True).json()['rows']]
-
-
-def default_geometry_id(column_id):
-    '''
-    Returns default test point for the column_id.
-    '''
-    if column_id == 'whosonfirst.wof_disputed_geom':
-        return 'CDB_LatLng(33.78, 76.57)'
-    elif column_id == 'whosonfirst.wof_marinearea_geom':
-        return 'CDB_LatLng(43.33, -68.47)'
-    elif column_id in ('us.census.tiger.school_district_elementary',
-                       'us.census.tiger.school_district_secondary',
-                       'us.census.tiger.school_district_elementary_clipped',
-                       'us.census.tiger.school_district_secondary_clipped'):
-        return 'CDB_LatLng(40.7025, -73.7067)'
-    elif column_id.startswith('es.ine'):
-        return 'CDB_LatLng(42.8226119029222, -2.51141249535454)'
-    elif column_id.startswith('us.zillow'):
-        return 'CDB_LatLng(28.3305906291771, -81.3544048197256)'
-    else:
-        return 'CDB_LatLng(40.7, -73.9)'
-
-
-def default_point(column_id):
-    '''
-    Returns default test point for the column_id.
-    '''
-    if column_id == 'whosonfirst.wof_disputed_geom':
-        return 'CDB_LatLng(33.78, 76.57)'
-    elif column_id == 'whosonfirst.wof_marinearea_geom':
-        return 'CDB_LatLng(43.33, -68.47)'
-    elif column_id in ('us.census.tiger.school_district_elementary',
-                       'us.census.tiger.school_district_secondary',
-                       'us.census.tiger.school_district_elementary_clipped',
-                       'us.census.tiger.school_district_secondary_clipped'):
-        return 'CDB_LatLng(40.7025, -73.7067)'
-    elif column_id.startswith('uk'):
-        if 'WA' in column_id:
-            return 'CDB_LatLng(51.46844551219723, -3.184833526611328)'
-        else:
-            return 'CDB_LatLng(51.51461834694225, -0.08883476257324219)'
-    elif column_id.startswith('es'):
-        return 'CDB_LatLng(42.8226119029222, -2.51141249535454)'
-    elif column_id.startswith('us.zillow'):
-        return 'CDB_LatLng(28.3305906291771, -81.3544048197256)'
-    elif column_id.startswith('mx.'):
-        return 'CDB_LatLng(19.41347699386547, -99.17019367218018)'
-    else:
-        return 'CDB_LatLng(40.7, -73.9)'
-
-
-def default_area(column_id):
-    '''
-    Returns default test area for the column_id
-    '''
-    point = default_point(column_id)
-    area = 'ST_Transform(ST_Buffer(ST_Transform({point}, 3857), 1000), 4326)'.format(
-        point=point)
-    return area
-
-@parameterized(US_CENSUS_MEASURE_COLUMNS)
-def test_get_us_census_measure_points(name):
-    print 'test_get_us_census_measure_points, ', name
-    resp = query('''
-SELECT * FROM {schema}OBS_GetUSCensusMeasure({point}, '{name}')
-                 '''.format(name=name.replace("'", "''"),
-                            schema='cdb_observatory.' if USE_SCHEMA else '',
-                            point=default_point('')))
-    assert_equal(resp.status_code, 200)
-    rows = resp.json()['rows']
-    assert_equal(1, len(rows))
-    assert_is_not_none(rows[0].values()[0])
-
-
-@parameterized(MEASURE_COLUMNS)
-def test_get_measure_areas(column_id, point_only):
-    print 'test_get_measure_areas, ', column_id, point_only
-    if point_only:
-        return
-    resp = query('''
-SELECT * FROM {schema}OBS_GetMeasure({area}, '{column_id}')
-                 '''.format(column_id=column_id,
-                            schema='cdb_observatory.' if USE_SCHEMA else '',
-                            area=default_area(column_id)))
-    assert_equal(resp.status_code, 200)
-    rows = resp.json()['rows']
-    assert_equal(1, len(rows))
-    assert_is_not_none(rows[0].values()[0])
-
-
-@parameterized(MEASURE_COLUMNS)
-def test_get_measure_points(column_id, point_only):
-    print 'test_get_measure_points, ', column_id, point_only
-    resp = query('''
-SELECT * FROM {schema}OBS_GetMeasure({point}, '{column_id}')
-                 '''.format(column_id=column_id,
-                            schema='cdb_observatory.' if USE_SCHEMA else '',
-                            point=default_point(column_id)))
-    assert_equal(resp.status_code, 200)
-    rows = resp.json()['rows']
-    assert_equal(1, len(rows))
-    assert_is_not_none(rows[0].values()[0])
-
-#@parameterized(CATEGORY_COLUMNS)
-#def test_get_category_areas(column_id):
-#    resp = query('''
-#SELECT * FROM {schema}OBS_GetCategory({area}, '{column_id}')
-#                 '''.format(column_id=column_id,
-#                            schema='cdb_observatory.' if USE_SCHEMA else '',
-#                            area=default_area(column_id)))
-#    assert_equal(resp.status_code, 200)
-#    rows = resp.json()['rows']
-#    assert_equal(1, len(rows))
-#    assert_is_not_none(rows[0].values()[0])
-
-@parameterized(CATEGORY_COLUMNS)
-def test_get_category_points(column_id):
-    print 'test_get_category_points, ', column_id
-    resp = query('''
-SELECT * FROM {schema}OBS_GetCategory({point}, '{column_id}')
-                 '''.format(column_id=column_id,
-                            schema='cdb_observatory.' if USE_SCHEMA else '',
-                            point=default_point(column_id)))
-    assert_equal(resp.status_code, 200)
-    rows = resp.json()['rows']
-    assert_equal(1, len(rows))
-    assert_is_not_none(rows[0].values()[0])
-
-#@parameterized(BOUNDARY_COLUMNS)
-#def test_get_boundaries_by_geometry(column_id):
-#    resp = query('''
-#SELECT * FROM {schema}OBS_GetBoundariesByGeometry({area}, '{column_id}')
-#                 '''.format(column_id=column_id,
-#                            schema='cdb_observatory.' if USE_SCHEMA else '',
-#                            area=default_area(column_id)))
-#    assert_equal(resp.status_code, 200)
-#    rows = resp.json()['rows']
-#    assert_equal(1, len(rows))
-#    assert_is_not_none(rows[0].values()[0])
-
-#@parameterized(BOUNDARY_COLUMNS)
-#def test_get_points_by_geometry(column_id):
-#    resp = query('''
-#SELECT * FROM {schema}OBS_GetPointsByGeometry({area}, '{column_id}')
-#                 '''.format(column_id=column_id,
-#                            schema='cdb_observatory.' if USE_SCHEMA else '',
-#                            area=default_area(column_id)))
-#    assert_equal(resp.status_code, 200)
-#    rows = resp.json()['rows']
-#    assert_equal(1, len(rows))
-#    assert_is_not_none(rows[0].values()[0])
-
-#@parameterized(BOUNDARY_COLUMNS)
-#def test_get_boundary_points(column_id):
-#    resp = query('''
-#SELECT * FROM {schema}OBS_GetBoundary({point}, '{column_id}')
-#                 '''.format(column_id=column_id,
-#                            schema='cdb_observatory.' if USE_SCHEMA else '',
-#                            point=default_point(column_id)))
-#    assert_equal(resp.status_code, 200)
-#    rows = resp.json()['rows']
-#    assert_equal(1, len(rows))
-#    assert_is_not_none(rows[0].values()[0])
-
-#@parameterized(BOUNDARY_COLUMNS)
-#def test_get_boundary_id(column_id):
-#    resp = query('''
-#SELECT * FROM {schema}OBS_GetBoundaryId({point}, '{column_id}')
-#                 '''.format(column_id=column_id,
-#                            schema='cdb_observatory.' if USE_SCHEMA else '',
-#                            point=default_point(column_id)))
-#    assert_equal(resp.status_code, 200)
-#    rows = resp.json()['rows']
-#    assert_equal(1, len(rows))
-#    assert_is_not_none(rows[0].values()[0])
-
-#@parameterized(BOUNDARY_COLUMNS)
-#def test_get_boundary_by_id(column_id):
-#    resp = query('''
-#SELECT * FROM {schema}OBS_GetBoundaryById({geometry_id}, '{column_id}')
-#                 '''.format(column_id=column_id,
-#                            schema='cdb_observatory.' if USE_SCHEMA else '',
-#                            geometry_id=default_geometry_id(column_id)))
-#    assert_equal(resp.status_code, 200)
-#    rows = resp.json()['rows']
-#    assert_equal(1, len(rows))
-#    assert_is_not_none(rows[0].values()[0])

scripts/generate_fixtures.py

@@ -1,213 +1,370 @@
+import os
+import psycopg2
+import subprocess
+
+DB_CONN = psycopg2.connect('postgres://{user}:{password}@{host}:{port}/{database}'.format(
+    user=os.environ.get('PGUSER', 'postgres'),
+    password=os.environ.get('PGPASSWORD', ''),
+    host=os.environ.get('PGHOST', 'localhost'),
+    port=os.environ.get('PGPORT', '5432'),
+    database=os.environ.get('PGDATABASE', 'postgres'),
+))
+CURSOR = DB_CONN.cursor()
+
+
+def query(q):
+    '''
+    Query the database.
+    '''
+    try:
+        CURSOR.execute(q)
+        return CURSOR
+    except:
+        DB_CONN.rollback()
+        raise
+
+
+def commit():
+    try:
+        DB_CONN.commit()
+    except:
+        DB_CONN.rollback()
+        raise
 
-from sqldumpr import Dumpr
 
 def get_tablename_query(column_id, boundary_id, timespan):
     """
         given a column_id, boundary-id (us.census.tiger.block_group), and
         timespan, give back the current table hash from the data observatory
     """
-    q = """
-        SELECT t.tablename, geoid_ct.colname colname
-        FROM obs_table t,
-             obs_column_table geoid_ct,
-             obs_column_table data_ct
-        WHERE
-             t.id = geoid_ct.table_id AND
-             t.id = data_ct.table_id AND
-             geoid_ct.column_id =
-        (SELECT source_id
-         FROM obs_column_to_column
-         WHERE target_id = '{boundary_id}'
-         AND reltype = 'geom_ref'
-        ) AND
-        data_ct.column_id = '{column_id}' AND
-        timespan = '{timespan}'
-    """.replace('\n','')
+    return """
+        SELECT numer_tablename, numer_geomref_colname, numer_tid,
+               geom_tablename, geom_geomref_colname, geom_tid
+        FROM observatory.obs_meta
+        WHERE numer_id = '{numer_id}' AND
+              geom_id = '{geom_id}' AND
+              numer_timespan = '{numer_timespan}'
+    """.format(numer_id=column_id,
+               geom_id=boundary_id,
+               numer_timespan=timespan)
 
-    return q.format(column_id=column_id,
-                    boundary_id=boundary_id,
-                    timespan=timespan)
 
-def select_star(tablename):
-    return "SELECT * FROM {}".format(tablename)
+METADATA_TABLES = ['obs_table', 'obs_column_table', 'obs_column', 'obs_column_tag',
+                   'obs_tag', 'obs_column_to_column', 'obs_dump_version', 'obs_meta',
+                   'obs_meta_numer', 'obs_meta_denom', 'obs_meta_geom',
+                   'obs_meta_timespan', 'obs_column_table_tile',
+                   'obs_column_table_tile_simple']
 
-cdb = Dumpr('observatory.cartodb.com','')
-
-metadata = ['obs_table', 'obs_column_table', 'obs_column', 'obs_column_tag',
-            'obs_tag', 'obs_column_to_column', 'obs_dump_version', ]
-
-fixtures = [
-    ('us.census.tiger.census_tract', 'us.census.tiger.census_tract', '2014'),
-    ('us.census.tiger.block_group', 'us.census.tiger.block_group', '2014'),
-    ('us.census.tiger.zcta5', 'us.census.tiger.zcta5', '2014'),
-    ('us.census.tiger.county', 'us.census.tiger.county', '2014'),
-    ('us.census.acs.B01003001', 'us.census.tiger.census_tract', '2010 - 2014'),
+FIXTURES = [
     ('us.census.acs.B01003001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B01001002_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B01001026_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B01002001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002003_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002004_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002006_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002012_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B05001006_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08006001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08006002_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08301010_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08006009_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08006011_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08006015_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08006017_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B09001001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B11001001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B14001001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B14001002_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B14001005_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B14001006_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B14001007_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B14001008_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003017_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003022_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003023_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B16001001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B16001002_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B16001003_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B17001001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B17001002_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19013001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19083001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19301001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25001001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25002003_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25004002_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25004004_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25058001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25071001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25075001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25075025_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
     ('us.census.acs.B01003001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B01001002', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B01001026', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B01002001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B03002003', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B03002004', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B03002006', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B03002012', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B03002005', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B03002008', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B03002009', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B03002002', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B11001001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B15003001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B15003017', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B15003019', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B15003020', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B15003021', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B15003022', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B15003023', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19013001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19301001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25001001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25002003', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25004002', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25004004', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25058001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25071001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25075001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25075025', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25081002', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B08134001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B08134002', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001002', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001003', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001004', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001005', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001006', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001007', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001008', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001009', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001010', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001011', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001012', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001013', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001014', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001015', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001016', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001017', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B01001002', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B01003001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B01001002', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B01001026', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B01002001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002003', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002004', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002006', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002012', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002005', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002008', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002009', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002002', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B11001001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003017', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003019', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003020', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003021', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003022', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003023', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19013001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19083001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19301001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25001001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25002003', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25004002', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25004004', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25058001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25071001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25075001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25075025', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25081002', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08134001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08134002', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08134008', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08134008', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08134010', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001002', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001003', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001004', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001005', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001006', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001007', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001008', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001009', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001010', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001011', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001012', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001013', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001014', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001015', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001016', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001017', 'us.census.tiger.census_tract', '2010 - 2014'),
     ('us.census.spielman_singleton_segments.X10', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.spielman_singleton_segments.X55', 'us.census.tiger.census_tract', '2010 - 2014'),
     ('us.zillow.AllHomes_Zhvi', 'us.census.tiger.zcta5', '2014-01'),
-    ('us.zillow.AllHomes_Zhvi', 'us.census.tiger.zcta5', '2016-03'),
-    ('whosonfirst.wof_country_geom', 'whosonfirst.wof_country_geom', '2016'),
-    ('us.census.tiger.zcta5_clipped', 'us.census.tiger.zcta5_clipped', '2014'),
-    ('us.census.tiger.block_group_clipped', 'us.census.tiger.block_group_clipped', '2014'),
+    ('us.zillow.AllHomes_Zhvi', 'us.census.tiger.zcta5', '2016-06'),
+    ('whosonfirst.wof_country_name', 'whosonfirst.wof_country_geom', '2016'),
+    ('us.census.acs.B01003001', 'us.census.tiger.zcta5_clipped', '2010 - 2014'),
+    ('us.census.acs.B01003001', 'us.census.tiger.block_group_clipped', '2010 - 2014'),
+    ('us.census.acs.B01003001', 'us.census.tiger.census_tract_clipped', '2010 - 2014'),
+    ('us.census.tiger.fullname', 'us.census.tiger.pointlm_geom', '2016'),
+    ('us.census.tiger.fullname', 'us.census.tiger.prisecroads_geom', '2016'),
+    ('us.census.tiger.name', 'us.census.tiger.county', '2015'),
+    ('us.census.tiger.name', 'us.census.tiger.block_group', '2015'),
 ]
 
-unique_tables = set()
+OUTFILE_PATH = os.path.join(os.path.dirname(__file__), '..',
+                            'src/pg/test/fixtures/load_fixtures.sql')
+DROPFILE_PATH = os.path.join(os.path.dirname(__file__), '..',
+                             'src/pg/test/fixtures/drop_fixtures.sql')
 
-for f in fixtures:
-    column_id, boundary_id, timespan = f
-    tablename_query = get_tablename_query(*f)
-    resp = cdb.query(tablename_query).json()['rows'][0]
-    tablename = resp['tablename']
-    colname = resp['colname']
-    table_colname = (tablename, colname, boundary_id, )
-    if table_colname not in unique_tables:
-        print table_colname
-        unique_tables.add(table_colname)
+def dump(cols, tablename, where=''):
 
-print unique_tables
+    with open(DROPFILE_PATH, 'a') as dropfile:
+        dropfile.write('DROP TABLE IF EXISTS observatory.{tablename};\n'.format(
+            tablename=tablename,
+        ))
 
-with open('src/pg/test/fixtures/load_fixtures.sql', 'w') as outfile:
-    with open('src/pg/test/fixtures/drop_fixtures.sql', 'w') as dropfiles:
-        outfile.write('SET client_min_messages TO WARNING;\n\set ECHO none\n')
-        dropfiles.write('SET client_min_messages TO WARNING;\n\set ECHO none\n')
-        for tablename in metadata:
-            cdb.dump(select_star(tablename), tablename, outfile, schema='observatory')
-            dropfiles.write('DROP TABLE IF EXISTS observatory.{};\n'.format(tablename))
-            print tablename
+    subprocess.check_call('pg_dump -x --section=pre-data -t observatory.{tablename} '
+                          ' | sed "s:SET search_path.*::" '
+                          ' | sed "s:CREATE TABLE :CREATE TABLE observatory.:" '
+                          ' | sed "s:ALTER TABLE.*OWNER.*::" '
+                          ' | sed "s:SET idle_in_transaction_session_timeout.*::" '
+                          ' >> {outfile}'.format(
+                              tablename=tablename,
+                              outfile=OUTFILE_PATH,
+                          ), shell=True)
 
-        for tablename, colname, boundary_id in unique_tables:
-            if 'zcta5' in boundary_id:
-                where = '\'11%\''
-                compare = 'LIKE'
-            elif 'whosonfirst' in boundary_id:
-                where = '(\'85632785\',\'85633051\',\'85633111\',\'85633147\',\'85633253\',\'85633267\')'
-                compare = 'IN'
-            else:
-                where = '\'36047%\''
-                compare = 'LIKE'
-            print ' '.join([select_star(tablename), "WHERE {}::text {} {}".format(colname, compare, where)])
-            cdb.dump(' '.join([select_star(tablename), "WHERE {}::text {} {}".format(colname, compare, where)]),
-                     tablename, outfile, schema='observatory')
-            dropfiles.write('DROP TABLE IF EXISTS observatory.{};\n'.format(tablename))
+    with open(OUTFILE_PATH, 'a') as outfile:
+        outfile.write('COPY observatory."{}" FROM stdin WITH CSV HEADER;\n'.format(tablename))
+
+    subprocess.check_call('''
+      psql -c "COPY (SELECT {cols} \
+      FROM observatory.{tablename} {where}) \
+      TO STDOUT WITH CSV HEADER" >> {outfile}'''.format(
+          cols=cols,
+          tablename=tablename,
+          where=where,
+          outfile=OUTFILE_PATH,
+      ), shell=True)
+
+    with open(OUTFILE_PATH, 'a') as outfile:
+        outfile.write('\\.\n\n')
 
 
-        outfile.write('''
-ALTER TABLE observatory.obs_table
-  ADD PRIMARY KEY (id);
-ALTER TABLE observatory.obs_column_table
-  ADD PRIMARY KEY (column_id, table_id);
-CREATE UNIQUE INDEX ON observatory.obs_column_table (table_id, column_id);
-CREATE UNIQUE INDEX ON observatory.obs_column_table (table_id, colname);
-ALTER TABLE observatory.obs_column
-  ADD PRIMARY KEY (id);
-ALTER TABLE observatory.obs_column_to_column
-  ADD PRIMARY KEY (source_id, target_id, reltype);
-CREATE UNIQUE INDEX ON observatory.obs_column_to_column (target_id, source_id, reltype);
-CREATE INDEX ON observatory.obs_column_to_column (reltype);
-ALTER TABLE observatory.obs_column_tag
-  ADD PRIMARY KEY (column_id, tag_id);
-CREATE UNIQUE INDEX ON observatory.obs_column_tag (tag_id, column_id);
-ALTER TABLE observatory.obs_tag
-  ADD PRIMARY KEY (id);
-CREATE INDEX ON observatory.obs_tag (type);
+def main():
+    unique_tables = set()
 
-VACUUM ANALYZE observatory.obs_table;
-VACUUM ANALYZE observatory.obs_column_table;
-VACUUM ANALYZE observatory.obs_column;
-VACUUM ANALYZE observatory.obs_column_to_column;
-VACUUM ANALYZE observatory.obs_column_tag;
-VACUUM ANALYZE observatory.obs_tag;
+    for f in FIXTURES:
+        column_id, boundary_id, timespan = f
+        tablename_query = get_tablename_query(column_id, boundary_id, timespan)
+        resp = query(tablename_query).fetchone()
+        if resp:
+            numer_tablename, numer_colname, numer_table_id = resp[0:3]
+            geom_tablename, geom_colname, geom_table_id = resp[3:6]
+        else:
+            raise Exception("Could not find table for {}, {}, {}".format(
+                column_id, boundary_id, timespan))
+        numer = (numer_tablename, numer_colname, numer_table_id, )
+        geom = (geom_tablename, geom_colname, geom_table_id, )
+        if numer not in unique_tables:
+            print(numer)
+            unique_tables.add(numer)
+        if geom not in unique_tables:
+            print(geom)
+            unique_tables.add(geom)
 
-CREATE TABLE observatory.obs_meta AS
-SELECT numer_c.id numer_id,
-       denom_c.id denom_id,
-       geom_c.id geom_id,
-       MAX(numer_c.name) numer_name,
-       MAX(denom_c.name) denom_name,
-       MAX(geom_c.name) geom_name,
-       MAX(numer_c.description) numer_description,
-       MAX(denom_c.description) denom_description,
-       MAX(geom_c.description) geom_description,
-       MAX(numer_c.aggregate) numer_aggregate,
-       MAX(denom_c.aggregate) denom_aggregate,
-       MAX(geom_c.aggregate) geom_aggregate,
-       MAX(numer_c.type) numer_type,
-       MAX(denom_c.type) denom_type,
-       MAX(geom_c.type) geom_type,
-       MAX(numer_data_ct.colname) numer_colname,
-       MAX(denom_data_ct.colname) denom_colname,
-       MAX(geom_geom_ct.colname) geom_colname,
-       MAX(numer_geomref_ct.colname) numer_geomref_colname,
-       MAX(denom_geomref_ct.colname) denom_geomref_colname,
-       MAX(geom_geomref_ct.colname) geom_geomref_colname,
-       MAX(numer_t.tablename) numer_tablename,
-       MAX(denom_t.tablename) denom_tablename,
-       MAX(geom_t.tablename) geom_tablename,
-       MAX(numer_t.timespan) numer_timespan,
-       MAX(denom_t.timespan) denom_timespan,
-       MAX(numer_c.weight) numer_weight,
-       MAX(denom_c.weight) denom_weight,
-       MAX(geom_c.weight) geom_weight,
-       MAX(geom_t.timespan) geom_timespan,
-       MAX(geom_t.the_geom_webmercator)::geometry AS the_geom_webmercator,
-       ARRAY_AGG(DISTINCT s_tag.id) section_tags,
-       ARRAY_AGG(DISTINCT ss_tag.id) subsection_tags,
-       ARRAY_AGG(DISTINCT unit_tag.id) unit_tags
-FROM observatory.obs_column_table numer_data_ct,
-     observatory.obs_table numer_t,
-     observatory.obs_column_table numer_geomref_ct,
-     observatory.obs_column geomref_c,
-     observatory.obs_column_to_column geomref_c2c,
-     observatory.obs_column geom_c,
-     observatory.obs_column_table geom_geom_ct,
-     observatory.obs_column_table geom_geomref_ct,
-     observatory.obs_table geom_t,
-     observatory.obs_column_tag ss_ctag,
-     observatory.obs_tag ss_tag,
-     observatory.obs_column_tag s_ctag,
-     observatory.obs_tag s_tag,
-     observatory.obs_column_tag unit_ctag,
-     observatory.obs_tag unit_tag,
-     observatory.obs_column numer_c
-  LEFT JOIN (
-    observatory.obs_column_to_column denom_c2c
-    JOIN observatory.obs_column denom_c ON denom_c2c.target_id = denom_c.id
-    JOIN observatory.obs_column_table denom_data_ct ON denom_data_ct.column_id = denom_c.id
-    JOIN observatory.obs_table denom_t ON denom_data_ct.table_id = denom_t.id
-    JOIN observatory.obs_column_table denom_geomref_ct ON denom_geomref_ct.table_id = denom_t.id
-  ) ON denom_c2c.source_id = numer_c.id
-WHERE numer_c.id = numer_data_ct.column_id
-  AND numer_data_ct.table_id = numer_t.id
-  AND numer_t.id = numer_geomref_ct.table_id
-  AND numer_geomref_ct.column_id = geomref_c.id
-  AND geomref_c2c.reltype = 'geom_ref'
-  AND geomref_c.id = geomref_c2c.source_id
-  AND geom_c.id = geomref_c2c.target_id
-  AND geom_geomref_ct.column_id = geomref_c.id
-  AND geom_geomref_ct.table_id = geom_t.id
-  AND geom_geom_ct.column_id = geom_c.id
-  AND geom_geom_ct.table_id = geom_t.id
-  AND geom_c.type ILIKE 'geometry'
-  AND numer_c.type NOT ILIKE 'geometry'
-  AND numer_t.id != geom_t.id
-  AND numer_c.id != geomref_c.id
-  AND unit_tag.type = 'unit'
-  AND ss_tag.type = 'subsection'
-  AND s_tag.type = 'section'
-  AND unit_ctag.column_id = numer_c.id
-  AND unit_ctag.tag_id = unit_tag.id
-  AND ss_ctag.column_id = numer_c.id
-  AND ss_ctag.tag_id = ss_tag.id
-  AND s_ctag.column_id = numer_c.id
-  AND s_ctag.tag_id = s_tag.id
-  AND (denom_c2c.reltype = 'denominator' OR denom_c2c.reltype IS NULL)
-  AND (denom_geomref_ct.column_id = geomref_c.id OR denom_geomref_ct.column_id IS NULL)
-  AND (denom_t.timespan = numer_t.timespan OR denom_t.timespan IS NULL)
-GROUP BY numer_c.id, denom_c.id, geom_c.id,
-         numer_t.id, denom_t.id, geom_t.id;
-        ''')
+    print unique_tables
 
-        dropfiles.write('''
-DROP TABLE IF EXISTS observatory.obs_meta;
-                        ''')
+    with open(OUTFILE_PATH, 'w') as outfile:
+        outfile.write('SET client_min_messages TO WARNING;\n\\set ECHO none\n')
+        outfile.write('CREATE SCHEMA IF NOT EXISTS observatory;\n\n')
+
+    with open(DROPFILE_PATH, 'w') as dropfile:
+        dropfile.write('SET client_min_messages TO WARNING;\n\\set ECHO none\n')
+
+    for tablename in METADATA_TABLES:
+        print(tablename)
+        if tablename == 'obs_meta':
+            where = "WHERE " + " OR ".join([
+                "(numer_id, geom_id, numer_timespan) = ('{}', '{}', '{}')".format(
+                    numer_id, geom_id, timespan)
+                for numer_id, geom_id, timespan in FIXTURES
+            ])
+        elif tablename == 'obs_meta_numer':
+            where = "WHERE " + " OR ".join([
+                "numer_id IN ('{}', '{}')".format(numer_id, geom_id)
+                for numer_id, geom_id, timespan in FIXTURES
+            ])
+        elif tablename == 'obs_meta_denom':
+            where = "WHERE " + " OR ".join([
+                "denom_id IN ('{}', '{}')".format(numer_id, geom_id)
+                for numer_id, geom_id, timespan in FIXTURES
+            ])
+        elif tablename == 'obs_meta_geom':
+            where = "WHERE " + " OR ".join([
+                "geom_id IN ('{}', '{}')".format(numer_id, geom_id)
+                for numer_id, geom_id, timespan in FIXTURES
+            ])
+        elif tablename == 'obs_meta_timespan':
+            where = "WHERE " + " OR ".join([
+                "timespan_id = ('{}')".format(timespan)
+                for numer_id, geom_id, timespan in FIXTURES
+            ])
+        elif tablename == 'obs_column':
+            where = "WHERE " + " OR ".join([
+                "id IN ('{}', '{}')".format(numer_id, geom_id)
+                for numer_id, geom_id, timespan in FIXTURES
+            ])
+        elif tablename == 'obs_column_tag':
+            where = "WHERE " + " OR ".join([
+                "column_id IN ('{}', '{}')".format(numer_id, geom_id)
+                for numer_id, geom_id, timespan in FIXTURES
+            ])
+        elif tablename in ('obs_column_table', 'obs_column_table_tile',
+                           'obs_column_table_tile_simple'):
+            where = '''WHERE table_id IN ({table_ids}) AND
+              (column_id IN ({numer_ids}) OR column_id IN ({geom_ids}))
+                    '''.format(
+                        numer_ids=','.join(["'{}'".format(x) for x, _, _ in FIXTURES]),
+                        geom_ids=','.join(["'{}'".format(x) for _, x, _ in FIXTURES]),
+                        table_ids=','.join(["'{}'".format(x) for _, _, x in unique_tables])
+                    )
+        elif tablename == 'obs_column_to_column':
+            where = "WHERE " + " OR ".join([
+                "source_id IN ('{}', '{}') OR target_id IN ('{}', '{}')".format(
+                    numer_id, geom_id, numer_id, geom_id)
+                for numer_id, geom_id, timespan in FIXTURES
+            ])
+        elif tablename == 'obs_table':
+            where = 'WHERE timespan IN ({timespans}) ' \
+                    'OR id IN ({table_ids}) '.format(
+                        timespans=','.join(["'{}'".format(x) for _, _, x in FIXTURES]),
+                        table_ids=','.join(["'{}'".format(x) for _, _, x in unique_tables])
+                    )
+        else:
+            where = ''
+        dump('*', tablename, where)
+
+    for tablename, colname, table_id in unique_tables:
+        if 'zcta5' in table_id or 'zillow_zip' in table_id:
+            where = '\'11%\''
+            compare = 'LIKE'
+        elif 'pri_sec_roads' in table_id or 'point_landmark' in table_id:
+            dump('*', tablename, 'WHERE geom && ST_MakeEnvelope(-74,40.69,-73.9,40.72, 4326)')
+            continue
+        elif 'whosonfirst' in table_id:
+            where = '(\'85632785\',\'85633051\',\'85633111\',\'85633147\',\'85633253\',\'85633267\')'
+            compare = 'IN'
+        else:
+            where = '\'36047%\''
+            compare = 'LIKE'
+        print ' '.join(['*', tablename, "WHERE {}::text {} {}".format(colname, compare, where)])
+        dump('*', tablename, "WHERE {}::text {} {}".format(colname, compare, where))
+
+if __name__ == '__main__':
+    main()

src/pg/observatory.control

@@ -1,5 +1,5 @@
 comment = 'CartoDB Observatory backend extension'
-default_version = '1.0.3'
-requires = 'postgis, postgres_fdw'
+default_version = '1.3.1'
+requires = 'postgis'
 superuser = true
 schema = cdb_observatory

src/pg/sql/15_fdw_utilities.sql

@@ -1,67 +0,0 @@
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_ConnectRemoteTable(fdw_name text, schema_name text, user_dbname text, user_hostname text, username text, user_tablename text, user_schema text)
-RETURNS void
-AS $$
-DECLARE
-  row record;
-  option record;
-  connection_str json;
-BEGIN
-  -- Build connection string
-  connection_str := '{"server":{"extensions":"postgis", "dbname":"'
-    || user_dbname ||'", "host":"' || user_hostname ||'", "port":"5432"}, "users":{"public"'
-    || ':{"user":"' || username ||'", "password":""} } }';
-
-  -- This function tries to be as idempotent as possible, by not creating anything more than once
-  -- (not even using IF NOT EXIST to avoid throwing warnings)
-  IF NOT EXISTS ( SELECT * FROM pg_extension WHERE extname = 'postgres_fdw') THEN
-    CREATE EXTENSION postgres_fdw;
-  END IF;
-  -- Create FDW first if it does not exist
-  IF NOT EXISTS ( SELECT * FROM pg_foreign_server WHERE srvname = fdw_name)
-    THEN
-    EXECUTE FORMAT('CREATE SERVER %I FOREIGN DATA WRAPPER postgres_fdw', fdw_name);
-  END IF;
-
-  -- Set FDW settings
-  FOR row IN SELECT p.key, p.value from lateral json_each_text(connection_str->'server') p
-    LOOP
-      IF NOT EXISTS (WITH a AS (select split_part(unnest(srvoptions), '=', 1) as options from pg_foreign_server where srvname=fdw_name) SELECT * from a where options = row.key)
-        THEN
-        EXECUTE FORMAT('ALTER SERVER %I OPTIONS (ADD %I %L)', fdw_name, row.key, row.value);
-      ELSE
-        EXECUTE FORMAT('ALTER SERVER %I OPTIONS (SET %I %L)', fdw_name, row.key, row.value);
-      END IF;
-    END LOOP;
-
-    -- Create user mappings
-    FOR row IN SELECT p.key, p.value from lateral json_each(connection_str->'users') p LOOP
-      -- Check if entry on pg_user_mappings exists
-      IF NOT EXISTS ( SELECT * FROM pg_user_mappings WHERE srvname = fdw_name AND usename = row.key ) THEN
-        EXECUTE FORMAT ('CREATE USER MAPPING FOR %I SERVER %I', row.key, fdw_name);
-      END IF;
-
-      -- Update user mapping settings
-      FOR option IN SELECT o.key, o.value from lateral json_each_text(row.value) o LOOP
-        IF NOT EXISTS (WITH a AS (select split_part(unnest(umoptions), '=', 1) as options from pg_user_mappings WHERE srvname = fdw_name AND usename = row.key) SELECT * from a where options = option.key) THEN
-          EXECUTE FORMAT('ALTER USER MAPPING FOR %I SERVER %I OPTIONS (ADD %I %L)', row.key, fdw_name, option.key, option.value);
-        ELSE
-          EXECUTE FORMAT('ALTER USER MAPPING FOR %I SERVER %I OPTIONS (SET %I %L)', row.key, fdw_name, option.key, option.value);
-        END IF;
-      END LOOP;
-    END LOOP;
-
-    -- Create schema if it does not exist.
-    IF NOT EXISTS ( SELECT * from pg_namespace WHERE nspname=fdw_name) THEN
-      EXECUTE FORMAT ('CREATE SCHEMA %I', fdw_name);
-    END IF;
-
-    -- Bring the remote cdb_tablemetadata
-    IF NOT EXISTS ( SELECT * FROM PG_CLASS WHERE relnamespace = (SELECT oid FROM pg_namespace WHERE nspname=fdw_name) and relname='cdb_tablemetadata') THEN
-      EXECUTE FORMAT ('CREATE FOREIGN TABLE %I.cdb_tablemetadata (tabname text, updated_at timestamp with time zone) SERVER %I OPTIONS (table_name ''cdb_tablemetadata_text'', schema_name ''public'', updatable ''false'')', fdw_name, fdw_name);
-    END IF;
-
-    -- Import target table
-    EXECUTE FORMAT ('IMPORT FOREIGN SCHEMA %I LIMIT TO (%I) from SERVER %I INTO %I', user_schema, user_tablename, fdw_name, schema_name);
-
-END;
-$$ LANGUAGE PLPGSQL;

src/pg/sql/40_observatory_utility.sql

@@ -203,3 +203,31 @@ BEGIN
   RETURN result;
 END;
 $$ LANGUAGE plpgsql;
+
+
+-- Function we can call to raise an exception in the midst of a SQL statement
+CREATE OR REPLACE FUNCTION cdb_observatory._OBS_RaiseNotice(
+    message TEXT
+) RETURNS TEXT
+AS $$
+BEGIN
+  RAISE NOTICE '%', message;
+  RETURN NULL;
+END;
+$$ LANGUAGE plpgsql;
+
+
+-- Create a function that always returns the first non-NULL item
+CREATE OR REPLACE FUNCTION cdb_observatory.first_agg ( anyelement, anyelement )
+RETURNS anyelement LANGUAGE SQL IMMUTABLE STRICT AS $$
+        SELECT $1;
+$$;
+
+DROP AGGREGATE IF EXISTS cdb_observatory.FIRST (anyelement);
+
+-- And then wrap an aggregate around it
+CREATE AGGREGATE cdb_observatory.FIRST (
+        sfunc    = cdb_observatory.first_agg,
+        basetype = anyelement,
+        stype    = anyelement
+);

src/pg/sql/42_observatory_exploration.sql

@@ -1,4 +1,4 @@
--- return a table that contains a string match based on input
+
 -- TODO: implement search for timespan
 
 CREATE OR REPLACE FUNCTION cdb_observatory._OBS_SearchTables(
@@ -120,3 +120,377 @@ BEGIN
   RETURN;
 END
 $$ LANGUAGE plpgsql;
+
+-- Functions the interface works from to identify available numerators,
+-- denominators, geometries, and timespans
+
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetAvailableNumerators(
+  bounds GEOMETRY DEFAULT NULL,
+  filter_tags TEXT[] DEFAULT NULL,
+  denom_id TEXT DEFAULT NULL,
+  geom_id TEXT DEFAULT NULL,
+  timespan TEXT DEFAULT NULL
+) RETURNS TABLE (
+  numer_id TEXT,
+  numer_name TEXT,
+  numer_description TEXT,
+  numer_weight NUMERIC,
+  numer_license TEXT,
+  numer_source TEXT,
+  numer_type TEXT,
+  numer_aggregate TEXT,
+  numer_extra JSONB,
+  numer_tags JSONB,
+  valid_denom BOOLEAN,
+  valid_geom BOOLEAN,
+  valid_timespan BOOLEAN
+) AS $$
+DECLARE
+  geom_clause TEXT;
+BEGIN
+  filter_tags := COALESCE(filter_tags, (ARRAY[])::TEXT[]);
+  denom_id := COALESCE(denom_id, '');
+  geom_id := COALESCE(geom_id, '');
+  timespan := COALESCE(timespan, '');
+  IF bounds IS NULL THEN
+    geom_clause := '';
+  ELSE
+    geom_clause := 'ST_Intersects(the_geom, $5) AND';
+  END IF;
+  RETURN QUERY
+  EXECUTE
+  format($string$
+    SELECT numer_id::TEXT,
+           numer_name::TEXT,
+           numer_description::TEXT,
+           numer_weight::NUMERIC,
+           NULL::TEXT license,
+           NULL::TEXT source,
+           numer_type numer_type,
+           numer_aggregate numer_aggregate,
+           numer_extra::JSONB numer_extra,
+           numer_tags numer_tags,
+    $1 = ANY(denoms) valid_denom,
+    $2 = ANY(geoms) valid_geom,
+    $3 = ANY(timespans) valid_timespan
+    FROM observatory.obs_meta_numer
+    WHERE %s (numer_tags ?& $4 OR CARDINALITY($4) = 0)
+  $string$, geom_clause)
+  USING denom_id, geom_id, timespan, filter_tags, bounds;
+  RETURN;
+END
+$$ LANGUAGE plpgsql;
+
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetAvailableDenominators(
+  bounds GEOMETRY DEFAULT NULL,
+  filter_tags TEXT[] DEFAULT NULL,
+  numer_id TEXT DEFAULT NULL,
+  geom_id TEXT DEFAULT NULL,
+  timespan TEXT DEFAULT NULL
+) RETURNS TABLE (
+  denom_id TEXT,
+  denom_name TEXT,
+  denom_description TEXT,
+  denom_weight NUMERIC,
+  denom_license TEXT,
+  denom_source TEXT,
+  denom_type TEXT,
+  denom_aggregate TEXT,
+  denom_extra JSONB,
+  denom_tags JSONB,
+  valid_numer BOOLEAN,
+  valid_geom BOOLEAN,
+  valid_timespan BOOLEAN
+) AS $$
+DECLARE
+  geom_clause TEXT;
+BEGIN
+  filter_tags := COALESCE(filter_tags, (ARRAY[])::TEXT[]);
+  numer_id := COALESCE(numer_id, '');
+  geom_id := COALESCE(geom_id, '');
+  timespan := COALESCE(timespan, '');
+  IF bounds IS NULL THEN
+    geom_clause := '';
+  ELSE
+    geom_clause := 'ST_Intersects(the_geom, $5) AND';
+  END IF;
+  RETURN QUERY
+  EXECUTE
+  format($string$
+    SELECT denom_id::TEXT,
+           denom_name::TEXT,
+           denom_description::TEXT,
+           denom_weight::NUMERIC,
+           NULL::TEXT license,
+           NULL::TEXT source,
+           denom_type::TEXT,
+           denom_aggregate::TEXT,
+           denom_extra::JSONB,
+           denom_tags::JSONB,
+    $1 = ANY(numers) valid_numer,
+    $2 = ANY(geoms) valid_geom,
+    $3 = ANY(timespans) valid_timespan
+    FROM observatory.obs_meta_denom
+    WHERE %s (denom_tags ?& $4 OR CARDINALITY($4) = 0)
+  $string$, geom_clause)
+  USING numer_id, geom_id, timespan, filter_tags, bounds;
+  RETURN;
+END
+$$ LANGUAGE plpgsql;
+
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetAvailableGeometries(
+  bounds GEOMETRY DEFAULT NULL,
+  filter_tags TEXT[] DEFAULT NULL,
+  numer_id TEXT DEFAULT NULL,
+  denom_id TEXT DEFAULT NULL,
+  timespan TEXT DEFAULT NULL
+) RETURNS TABLE (
+  geom_id TEXT,
+  geom_name TEXT,
+  geom_description TEXT,
+  geom_weight NUMERIC,
+  geom_aggregate TEXT,
+  geom_license TEXT,
+  geom_source TEXT,
+  valid_numer BOOLEAN,
+  valid_denom BOOLEAN,
+  valid_timespan BOOLEAN,
+  score NUMERIC,
+  numtiles BIGINT,
+  notnull_percent NUMERIC,
+  numgeoms NUMERIC,
+  percentfill NUMERIC,
+  estnumgeoms NUMERIC,
+  meanmediansize NUMERIC
+) AS $$
+DECLARE
+  geom_clause TEXT;
+BEGIN
+  filter_tags := COALESCE(filter_tags, (ARRAY[])::TEXT[]);
+  numer_id := COALESCE(numer_id, '');
+  denom_id := COALESCE(denom_id, '');
+  timespan := COALESCE(timespan, '');
+  IF bounds IS NULL THEN
+    geom_clause := '';
+  ELSE
+    geom_clause := 'ST_Intersects(the_geom, $5) AND';
+  END IF;
+  RETURN QUERY
+  EXECUTE
+  format($string$
+    WITH available_geoms AS (
+      SELECT geom_id::TEXT,
+             geom_name::TEXT,
+             geom_description::TEXT,
+             geom_weight::NUMERIC,
+             NULL::TEXT geom_aggregate,
+             NULL::TEXT license,
+             NULL::TEXT source,
+      $1 = ANY(numers) valid_numer,
+      $2 = ANY(denoms) valid_denom,
+      $3 = ANY(timespans) valid_timespan
+      FROM observatory.obs_meta_geom
+      WHERE %s (geom_tags ?& $4 OR CARDINALITY($4) = 0)
+    ), scores AS (
+      SELECT * FROM cdb_observatory._OBS_GetGeometryScores($5,
+        (SELECT ARRAY_AGG(geom_id) FROM available_geoms)
+      )
+    ) SELECT available_geoms.*, score, numtiles, notnull_percent, numgeoms,
+             percentfill, estnumgeoms, meanmediansize
+      FROM available_geoms, scores
+      WHERE available_geoms.geom_id = scores.column_id
+  $string$, geom_clause)
+  USING numer_id, denom_id, timespan, filter_tags, bounds;
+  RETURN;
+END
+$$ LANGUAGE plpgsql;
+
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetAvailableTimespans(
+  bounds GEOMETRY DEFAULT NULL,
+  filter_tags TEXT[] DEFAULT NULL,
+  numer_id TEXT DEFAULT NULL,
+  denom_id TEXT DEFAULT NULL,
+  geom_id TEXT DEFAULT NULL
+) RETURNS TABLE (
+  timespan_id TEXT,
+  timespan_name TEXT,
+  timespan_description TEXT,
+  timespan_weight NUMERIC,
+  timespan_aggregate TEXT,
+  timespan_license TEXT,
+  timespan_source TEXT,
+  valid_numer BOOLEAN,
+  valid_denom BOOLEAN,
+  valid_geom BOOLEAN
+) AS $$
+DECLARE
+  geom_clause TEXT;
+BEGIN
+  filter_tags := COALESCE(filter_tags, (ARRAY[])::TEXT[]);
+  numer_id := COALESCE(numer_id, '');
+  denom_id := COALESCE(denom_id, '');
+  geom_id := COALESCE(geom_id, '');
+  IF bounds IS NULL THEN
+    geom_clause := '';
+  ELSE
+    geom_clause := 'ST_Intersects(the_geom, $5) AND';
+  END IF;
+  RETURN QUERY
+  EXECUTE
+  format($string$
+    SELECT timespan_id::TEXT,
+           timespan_name::TEXT,
+           timespan_description::TEXT,
+           timespan_weight::NUMERIC,
+           NULL::TEXT timespan_aggregate,
+           NULL::TEXT license,
+           NULL::TEXT source,
+    $1 = ANY(numers) valid_numer,
+    $2 = ANY(denoms) valid_denom,
+    $3 = ANY(geoms) valid_geom_id
+    FROM observatory.obs_meta_timespan
+    WHERE %s (timespan_tags ?& $4 OR CARDINALITY($4) = 0)
+  $string$, geom_clause)
+  USING numer_id, denom_id, geom_id, filter_tags, bounds;
+  RETURN;
+END
+$$ LANGUAGE plpgsql;
+
+
+-- Function below should replace SQL in
+-- https://github.com/CartoDB/cartodb/blob/ab465cb2918c917940e955963b0cd8a050c06600/lib/assets/javascripts/cartodb3/editor/layers/layer-content-views/analyses/data-observatory-metadata.js
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_LegacyBuilderMetadata(
+  aggregate_type TEXT DEFAULT NULL
+)
+RETURNS TABLE (
+  name TEXT,
+  subsection JSONB
+) AS $$
+DECLARE
+  aggregate_condition TEXT DEFAULT '';
+BEGIN
+  IF LOWER(aggregate_type) ILIKE 'sum' THEN
+    aggregate_condition := ' AND numer_aggregate IN (''sum'', ''median'', ''average'') ';
+  ELSIF aggregate_type IS NOT NULL THEN
+    aggregate_condition := format(' AND numer_aggregate ILIKE %L ', aggregate_type);
+  END IF;
+  RETURN QUERY
+  EXECUTE format($string$
+    WITH expanded AS (
+        SELECT JSONB_Build_Object('id', numer_id, 'name', numer_name) "column",
+               SUBSTR((sections).key, 9) section_id, (sections).value section_name,
+               SUBSTR((subsections).key, 12) subsection_id, (subsections).value subsection_name
+        FROM (
+          SELECT numer_id, numer_name,
+                 jsonb_each_text(numer_tags) as sections,
+                 jsonb_each_text as subsections
+          FROM (SELECT numer_id, numer_name, numer_tags,
+                       jsonb_each_text(numer_tags)
+                FROM cdb_observatory.obs_getavailablenumerators()
+                WHERE numer_weight > 0 %s
+               ) foo
+        ) bar
+        WHERE (sections).key LIKE 'section/%%'
+          AND (subsections).key LIKE 'subsection/%%'
+      ), grouped_by_subsections AS (
+        SELECT JSONB_Agg(JSONB_Build_Object('f1', "column")) AS columns,
+               section_id, section_name, subsection_id, subsection_name
+        FROM expanded
+        GROUP BY section_id, section_name, subsection_id, subsection_name
+      )
+    SELECT section_name as name, JSONB_Agg(
+      JSONB_Build_Object(
+        'f1', JSONB_Build_Object(
+          'name', subsection_name,
+          'id', subsection_id,
+          'columns', columns
+        )
+      )
+    ) as subsection
+    FROM grouped_by_subsections
+    GROUP BY section_name
+  $string$, aggregate_condition);
+  RETURN;
+END
+$$ LANGUAGE plpgsql;
+
+
+CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetGeometryScores(
+  bounds Geometry(Geometry, 4326) DEFAULT NULL,
+  filter_geom_ids TEXT[] DEFAULT NULL,
+  desired_num_geoms INTEGER DEFAULT NULL
+) RETURNS TABLE (
+  score NUMERIC,
+  numtiles BIGINT,
+  table_id TEXT,
+  column_id TEXT,
+  notnull_percent NUMERIC,
+  numgeoms NUMERIC,
+  percentfill NUMERIC,
+  estnumgeoms NUMERIC,
+  meanmediansize NUMERIC
+) AS $$
+BEGIN
+  IF desired_num_geoms IS NULL THEN
+    desired_num_geoms := 3000;
+  END IF;
+  filter_geom_ids := COALESCE(filter_geom_ids, (ARRAY[])::TEXT[]);
+  -- Very complex geometries simply fail.  For a boundary check, we can
+  -- comfortably get away with the simplicity of an envelope
+  IF ST_Npoints(bounds) > 10000 THEN
+    bounds := ST_Envelope(bounds);
+  END IF;
+  RETURN QUERY
+  EXECUTE $string$
+    WITH clipped_geom AS (
+      SELECT column_id, table_id
+        , CASE WHEN $1 IS NOT NULL THEN ST_Clip(tile, $1, True) -- -20
+               ELSE tile END clipped_tile
+        , tile
+      FROM observatory.obs_column_table_tile_simple
+      WHERE ($1 IS NULL OR ST_Intersects($1, tile))
+        AND (column_id = ANY($2) OR cardinality($2) = 0)
+    ), clipped_geom_countagg AS (
+      SELECT column_id, table_id
+        , BOOL_AND(ST_BandIsNoData(clipped_tile, 1)) nodata
+        , ST_CountAgg(clipped_tile, 1, False)::Numeric pixels -- -10
+      FROM clipped_geom
+      GROUP BY column_id, table_id
+    ), clipped_geom_reagg AS (
+      SELECT COUNT(*)::BIGINT cnt, a.column_id, a.table_id,
+             cdb_observatory.FIRST(nodata) first_nodata,
+             cdb_observatory.FIRST(pixels) first_pixel,
+             cdb_observatory.FIRST(tile) first_tile,
+             (ST_SummaryStatsAgg(clipped_tile, 1, False)).sum::Numeric sum_geoms, -- ND
+             (ST_SummaryStatsAgg(clipped_tile, 2, False)).mean::Numeric / 255 mean_fill --ND
+      FROM clipped_geom_countagg a, clipped_geom b
+      WHERE a.table_id = b.table_id
+        AND a.column_id = b.column_id
+      GROUP BY a.column_id, a.table_id
+    ), final AS (
+      SELECT
+        cnt, table_id, column_id
+        , NULL::Numeric AS notnull_percent
+        , (CASE WHEN first_nodata IS FALSE
+                THEN sum_geoms
+                ELSE COALESCE(ST_Value(first_tile, 1, ST_PointOnSurface($1)), 0)
+                  * (ST_Area($1) / ST_Area(ST_PixelAsPolygon(first_tile, 0, 0))
+                    * first_pixel) -- -20
+          END)::Numeric
+        AS numgeoms
+        , (CASE WHEN first_nodata IS FALSE
+                THEN mean_fill
+                ELSE COALESCE(ST_Value(first_tile, 2, ST_PointOnSurface($1))::Numeric / 255, 0) -- -2
+          END)::Numeric
+        AS percentfill
+        , null::numeric estnumgeoms
+        , null::numeric meanmediansize
+      FROM clipped_geom_reagg
+    ) SELECT
+      ((100.0 / (1+abs(log(0.0001 + $3) - log(0.0001 + numgeoms::Numeric)))) * percentfill)::Numeric
+      AS score, *
+      FROM final
+  $string$ USING bounds, filter_geom_ids, desired_num_geoms;
+  RETURN;
+END
+$$ LANGUAGE plpgsql IMMUTABLE;

src/pg/sql/44_observatory_geometries.sql

@@ -40,44 +40,11 @@ BEGIN
     RAISE EXCEPTION 'Invalid geometry type (%), expecting ''ST_Point''', ST_GeometryType(geom);
   END IF;
 
-  -- choose appropriate table based on time_span
-  IF time_span IS NULL
-  THEN
-    SELECT x.target_tables INTO target_table
-    FROM cdb_observatory._OBS_SearchTables(boundary_id,
-                                           time_span) As x(target_tables,
-                                                           timespans)
-    ORDER BY x.timespans DESC
-    LIMIT 1;
-  ELSE
-    -- TODO: modify for only one table returned instead of arbitrarily choosing
-    --       one with LIMIT 1 (could be conflict between clipped vs non-clipped
-    --       boundaries in the metadata tables)
-    SELECT x.target_tables INTO target_table
-    FROM cdb_observatory._OBS_SearchTables(boundary_id,
-                            time_span) As x(target_tables,
-                                            timespans)
-    WHERE x.timespans = time_span
-    LIMIT 1;
-  END IF;
-
-  -- if no tables are found, raise notice and return null
-  IF target_table IS NULL
-  THEN
-    --RAISE NOTICE 'No boundaries found for ''%'' in ''%''', ST_AsText(geom), boundary_id;
-    RETURN NULL::geometry;
-  END IF;
-
-  --RAISE NOTICE 'target_table: %', target_table;
-
   -- return the first boundary in intersections
-  EXECUTE format(
-    'SELECT the_geom
-     FROM observatory.%I
-     WHERE ST_Intersects($1, the_geom)
-     LIMIT 1', target_table)
-  INTO boundary
-  USING geom;
+  EXECUTE $query$
+    SELECT * FROM cdb_observatory._OBS_GetBoundariesByGeometry($1, $2, $3) LIMIT 1
+  $query$ INTO boundary
+  USING geom, boundary_id, time_span;
 
   RETURN boundary;
 
@@ -111,67 +78,17 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetBoundaryId(
 RETURNS text
 AS $$
 DECLARE
-  output_id text;
-  target_table text;
-  geoid_colname text;
+  result TEXT;
 BEGIN
 
-  -- If not point, raise error
-  IF ST_GeometryType(geom) != 'ST_Point'
-  THEN
-    RAISE EXCEPTION 'Invalid geometry type (%), expecting ''ST_Point''', ST_GeometryType(geom);
-  END IF;
-
-  -- choose appropriate table based on time_span
-  IF time_span IS NULL
-  THEN
-    SELECT x.target_tables INTO target_table
-    FROM cdb_observatory._OBS_SearchTables(boundary_id,
-                            time_span) As x(target_tables,
-                                                timespans)
-    ORDER BY x.timespans DESC
-    LIMIT 1;
-  ELSE
-    SELECT x.target_tables INTO target_table
-    FROM cdb_observatory._OBS_SearchTables(boundary_id,
-                            time_span) As x(target_tables,
-                                            timespans)
-    WHERE x.timespans = time_span
-    LIMIT 1;
-  END IF;
-
-  -- if no tables are found, raise notice and return null
-  IF target_table IS NULL
-  THEN
-    --RAISE NOTICE 'Warning: No boundaries found for ''%''', boundary_id;
-    RETURN NULL::text;
-  END IF;
-
-  EXECUTE
-    format('SELECT ct.colname
-              FROM observatory.obs_column_to_column c2c,
-                   observatory.obs_column_table ct,
-                   observatory.obs_table t
-             WHERE c2c.reltype = ''geom_ref''
-               AND ct.column_id = c2c.source_id
-               AND ct.table_id = t.id
-               AND t.tablename = %L'
-   , target_table)
-  INTO geoid_colname;
-
-  --RAISE NOTICE 'target_table: %, geoid_colname: %', target_table, geoid_colname;
-
-  -- return geometry id column value
-  EXECUTE format(
-    'SELECT %I::text
-     FROM observatory.%I
-     WHERE ST_Intersects($1, the_geom)
-     LIMIT 1', geoid_colname, target_table)
-  INTO output_id
-  USING geom;
-
-  RETURN output_id;
+  EXECUTE $query$
+    SELECT geom_refs FROM cdb_observatory._OBS_GetBoundariesByGeometry(
+      $1, $2, $3) LIMIT 1
+  $query$
+  INTO result
+  USING geom, boundary_id, time_span;
 
+  RETURN result;
 END;
 $$ LANGUAGE plpgsql;
 
@@ -203,35 +120,21 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetBoundaryById(
 RETURNS geometry(geometry, 4326)
 AS $$
 DECLARE
-  boundary geometry(geometry, 4326);
-  target_table text;
-  geoid_colname text;
-  geom_colname text;
+  result GEOMETRY;
 BEGIN
 
-  SELECT * INTO geoid_colname, target_table, geom_colname
-  FROM cdb_observatory._OBS_GetGeometryMetadata(boundary_id);
-
-  --RAISE NOTICE '%', target_table;
-
-  IF target_table IS NULL
-  THEN
-    --RAISE NOTICE 'No geometries found';
-    RETURN NULL::geometry;
-  END IF;
-
-  -- retrieve boundary
-  EXECUTE
-    format(
-    'SELECT %I
-     FROM observatory.%I
-     WHERE %I = $1
-     LIMIT 1', geom_colname, target_table, geoid_colname)
-  INTO boundary
-  USING geometry_id;
-
-  RETURN boundary;
+  EXECUTE $query$
+    SELECT (data->0->>'value')::Geometry
+    FROM cdb_observatory.OBS_GetData(
+      ARRAY[$1],
+      cdb_observatory.OBS_GetMeta(
+        ST_MakeEnvelope(-180, -90, 180, 90, 4326),
+        ('[{"geom_id": "' || $2 || '"}]')::JSON))
+  $query$
+  INTO result
+  USING geometry_id, boundary_id;
 
+  RETURN result;
 END;
 $$ LANGUAGE plpgsql;
 
@@ -244,49 +147,41 @@ CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetBoundariesByGeometry(
   geom geometry(Geometry, 4326),
   boundary_id text,
   time_span text DEFAULT NULL,
-  overlap_type text DEFAULT 'intersects')
-RETURNS TABLE(the_geom geometry, geom_refs text)
-AS $$
+  overlap_type text DEFAULT NULL)
+RETURNS TABLE (
+  the_geom geometry,
+  geom_refs text
+) AS $$
 DECLARE
-  boundary geometry(Geometry, 4326);
-  geom_colname text;
-  geoid_colname text;
-  target_table text;
+  meta JSON;
 BEGIN
-
+  overlap_type := COALESCE(overlap_type, 'intersects');
   -- check inputs
   IF lower(overlap_type) NOT IN ('contains', 'intersects', 'within')
   THEN
     -- recognized overlap type (map to ST_Contains, ST_Intersects, and ST_Within)
     RAISE EXCEPTION 'Overlap type ''%'' is not an accepted type (choose intersects, within, or contains)', overlap_type;
-  ELSIF ST_GeometryType(geom) NOT IN ('ST_Polygon', 'ST_MultiPolygon')
-  THEN
-      RAISE EXCEPTION 'Invalid geometry type (%), expecting ''ST_MultiPolygon'' or ''ST_Polygon''', ST_GeometryType(geom);
   END IF;
 
-  -- TODO: add timespan in search
-  -- TODO: add overlap info in search
-  SELECT * INTO geoid_colname, target_table, geom_colname
-  FROM cdb_observatory._OBS_GetGeometryMetadata(boundary_id);
+  EXECUTE $query$
+    SELECT cdb_observatory.OBS_GetMeta($1, JSON_Build_Array(JSON_Build_Object(
+                'geom_id', $2, 'geom_timespan', $3)))
+  $query$
+  INTO meta
+  USING geom, boundary_id, time_span;
 
-  -- if no tables are found, raise notice and return null
-  IF target_table IS NULL
-  THEN
-    --RAISE NOTICE 'No boundaries found for bounding box ''%'' in ''%''', ST_AsText(geom), boundary_id;
-    RETURN QUERY SELECT NULL::geometry, NULL::text;
+  IF meta->0->>'geom_id' IS NULL THEN
+    RETURN QUERY EXECUTE 'SELECT NULL::Geometry, NULL::Text LIMIT 0';
     RETURN;
   END IF;
 
-  --RAISE NOTICE 'target_table: %', target_table;
-
   -- return first boundary in intersections
-  RETURN QUERY
-  EXECUTE format(
-    'SELECT %I, %I::text
-     FROM observatory.%I
-     WHERE ST_%s($1, the_geom)
-     ', geom_colname, geoid_colname, target_table, overlap_type)
-  USING geom;
+  RETURN QUERY EXECUTE $query$
+    SELECT (data->0->>'value')::Geometry the_geom, data->0->>'geomref' geom_refs
+    FROM cdb_observatory.OBS_GetData(
+      ARRAY[($1, 1)::geomval], $2, False
+    )
+  $query$ USING geom, meta;
   RETURN;
 
 END;
@@ -318,7 +213,7 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetBoundariesByGeometry(
   geom geometry(Geometry, 4326),
   boundary_id text,
   time_span text DEFAULT NULL,
-  overlap_type text DEFAULT 'intersects')
+  overlap_type text DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text)
 AS $$
 BEGIN
@@ -364,7 +259,7 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetBoundariesByPointAndRadius(
   radius numeric, -- radius in meters
   boundary_id text,
   time_span text DEFAULT NULL,
-  overlap_type text DEFAULT 'intersects')
+  overlap_type text DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text)
 AS $$
 DECLARE
@@ -382,7 +277,8 @@ BEGIN
                FROM cdb_observatory._OBS_GetBoundariesByGeometry(
                         circle_boundary,
                         boundary_id,
-                        time_span);
+                        time_span,
+                        overlap_type);
   RETURN;
 END;
 $$ LANGUAGE plpgsql;
@@ -394,7 +290,7 @@ CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetPointsByGeometry(
   geom geometry(Geometry, 4326),
   boundary_id text,
   time_span text DEFAULT NULL,
-  overlap_type text DEFAULT 'intersects')
+  overlap_type text DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text)
 AS $$
 DECLARE
@@ -403,6 +299,7 @@ DECLARE
   geoid_colname text;
   target_table text;
 BEGIN
+  overlap_type := COALESCE(overlap_type, 'intersects');
 
   IF lower(overlap_type) NOT IN ('contains', 'within', 'intersects')
   THEN
@@ -412,27 +309,11 @@ BEGIN
     RAISE EXCEPTION 'Invalid geometry type (%), expecting ''ST_MultiPolygon'' or ''ST_Polygon''', ST_GeometryType(geom);
   END IF;
 
-  SELECT * INTO geoid_colname, target_table, geom_colname
-  FROM cdb_observatory._OBS_GetGeometryMetadata(boundary_id);
-
-  -- if no tables are found, raise notice and return null
-  IF target_table IS NULL
-  THEN
-    --RAISE NOTICE 'No boundaries found for bounding box ''%'' in ''%''', ST_AsText(geom), boundary_id;
-    RETURN QUERY SELECT NULL::geometry, NULL::text;
-    RETURN;
-  END IF;
-
-  --RAISE NOTICE 'target_table: %', target_table;
-
   -- return first boundary in intersections
-  RETURN QUERY
-  EXECUTE format(
-    'SELECT ST_PointOnSurface(%I) As %s, %I::text
-     FROM observatory.%I
-     WHERE ST_%s($1, the_geom)
-     ', geom_colname, geom_colname, geoid_colname, target_table, overlap_type)
-  USING geom;
+  RETURN QUERY EXECUTE $query$
+    SELECT ST_PointOnSurface(the_geom), geom_refs
+    FROM cdb_observatory._OBS_GetBoundariesByGeometry($1, $2)
+  $query$ USING geom, boundary_id;
   RETURN;
 
 END;
@@ -464,7 +345,7 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetPointsByGeometry(
   geom geometry(Geometry, 4326),
   boundary_id text,
   time_span text DEFAULT NULL,
-  overlap_type text DEFAULT 'intersects')
+  overlap_type text DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text)
 AS $$
 BEGIN
@@ -509,7 +390,7 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetPointsByPointAndRadius(
   radius numeric, -- radius in meters
   boundary_id text,
   time_span text DEFAULT NULL,
-  overlap_type text DEFAULT 'intersects')
+  overlap_type text DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text)
 AS $$
 DECLARE
@@ -532,44 +413,3 @@ BEGIN
   RETURN;
 END;
 $$ LANGUAGE plpgsql;
-
-
--- _OBS_GetGeometryMetadata()
--- TODO: add timespan in search
--- TODO: add choice of clipped versus not clipped
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetGeometryMetadata(boundary_id text)
-RETURNS table(geoid_colname text, target_table text, geom_colname text)
-AS $$
-BEGIN
-
-  RETURN QUERY
-  EXECUTE
-  format($string$
-    SELECT geoid_ct.colname::text As geoid_colname,
-           tablename::text,
-           geom_ct.colname::text As geom_colname
-    FROM observatory.obs_column_table As geoid_ct,
-         observatory.obs_table As geom_t,
-         observatory.obs_column_table As geom_ct,
-         observatory.obs_column As geom_c
-    WHERE geoid_ct.column_id
-       IN (
-         SELECT source_id
-         FROM observatory.obs_column_to_column
-         WHERE reltype = 'geom_ref'
-           AND target_id = '%s'
-         )
-      AND geoid_ct.table_id = geom_t.id AND
-          geom_t.id = geom_ct.table_id AND
-          geom_ct.column_id = geom_c.id AND
-          geom_c.type ILIKE 'geometry' AND
-          geom_c.id = '%s'
-    $string$, boundary_id, boundary_id);
-  RETURN;
-    --  AND geom_t.timespan = '%s' <-- put in requested year
-    -- TODO: filter by clipped vs. not so appropriate tablename are unique
-    --       so the limit 1 can be removed
-    RETURN;
-
-END;
-$$ LANGUAGE plpgsql;

src/pg/sql/50_table_level_functions.sql

@@ -1,119 +0,0 @@
-CREATE TYPE cdb_observatory.ds_fdw_metadata as (schemaname text, tabname text, servername text);
-CREATE TYPE cdb_observatory.ds_return_metadata as (colnames text[], coltypes text[]);
-
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_ConnectUserTable(username text, orgname text, user_db_role text, input_schema text, dbname text, host_addr text, table_name text)
-RETURNS cdb_observatory.ds_fdw_metadata
-AS $$
-DECLARE
-  fdw_server text;
-  fdw_import_schema text;
-  connection_str json;
-  import_foreign_schema_q text;
-  epoch_timestamp text;
-BEGIN
-
-  SELECT extract(epoch from now() at time zone 'utc')::int INTO epoch_timestamp;
-  fdw_server := 'fdw_server_' || username || '_' || epoch_timestamp;
-  fdw_import_schema:= fdw_server;
-
-  -- Import foreign table
-  EXECUTE FORMAT ('SELECT cdb_observatory._OBS_ConnectRemoteTable(%L, %L, %L, %L, %L, %L, %L)', fdw_server, fdw_import_schema, dbname, host_addr, user_db_role, table_name, input_schema);
-
-  RETURN (fdw_import_schema::text, table_name::text, fdw_server::text);
-
-EXCEPTION
-  WHEN others THEN
-    -- Disconnect user imported table. Delete schema and FDW server.
-    EXECUTE 'DROP FOREIGN TABLE IF EXISTS ' || fdw_import_schema || '.' || table_name;
-    EXECUTE 'DROP SCHEMA IF EXISTS ' || fdw_import_schema || ' CASCADE';
-    EXECUTE 'DROP SERVER IF EXISTS ' || fdw_server || ' CASCADE;';
-    RETURN (null, null, null);
-END;
-$$ LANGUAGE plpgsql SECURITY DEFINER;
-
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetReturnMetadata(username text, orgname text, function_name text, params json)
-RETURNS cdb_observatory.ds_return_metadata
-AS $$
-DECLARE
-  colnames text[];
-  coltypes text[];
-  requested_measures text[];
-  measure text;
-BEGIN
-
-  -- Simple mock, there should be real logic in here.
-
-  IF $3 NOT ILIKE 'GetMeasure' OR $3 IS NULL THEN
-    RAISE 'This function is not supported yet: %', $3;
-  END IF;
-
-  SELECT translate($4::json->>'tag_name','[]', '{}')::text[] INTO requested_measures;
-
-  FOREACH measure IN ARRAY requested_measures
-  LOOP
-    IF NOT measure ILIKE ANY (Array['total_pop', 'pop_16_over']::text[]) THEN
-      RAISE 'This measure is not supported yet: %', measure;
-    END IF;
-  SELECT array_append(colnames, measure) INTO colnames;
-  SELECT array_append(coltypes, 'double precision'::text) INTO coltypes;
-
-  END LOOP;
-
-  RETURN (colnames::text[], coltypes::text[]);
-END;
-$$ LANGUAGE plpgsql;
-
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_FetchJoinFdwTableData(username text, orgname text, table_schema text, table_name text, function_name text, params json)
-RETURNS SETOF record
-AS $$
-DECLARE
-  data_query text;
-  tag_name text[];
-  tag text;
-  tags_list text;
-  tags_query text;
-  rec RECORD;
-BEGIN
-    SELECT translate($6::json->>'tag_name','[]', '{}')::text[] INTO tag_name;
-    SELECT array_to_string(tag_name, ',') INTO tags_list;
-    tags_query := '';
-
-    FOREACH tag IN ARRAY tag_name
-    LOOP
-      SELECT tags_query || ' sum(' || tag || '/fraction)::double precision as ' || tag || ', ' INTO tags_query;
-
-    END LOOP;
-
-    -- Simple mock, there should be real logic in here.
-    data_query := '(WITH _areas AS(SELECT ST_Area(a.the_geom::geography)'
-        || '/ (1000 * 1000) as fraction, a.geoid, b.cartodb_id FROM '
-        || 'observatory.obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308 as a, '
-        || table_schema || '.' || table_name || ' AS b '
-        || 'WHERE b.the_geom && a.the_geom ), values AS (SELECT geoid, '
-        || tags_list
-        || ' FROM observatory.obs_1a098da56badf5f32e336002b0a81708c40d29cd ) '
-        || 'SELECT '
-        || tags_query
-        || ' cartodb_id::int FROM _areas, values '
-        || 'WHERE values.geoid = _areas.geoid GROUP BY cartodb_id);';
-
-
-    FOR rec IN EXECUTE data_query
-    LOOP
-        RETURN NEXT rec;
-    END LOOP;
-    RETURN;
-END;
-$$ LANGUAGE plpgsql SECURITY DEFINER;
-
-
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_DisconnectUserTable(username text, orgname text, table_schema text, table_name text, servername text)
-RETURNS boolean
-AS $$
-BEGIN
-    EXECUTE 'DROP FOREIGN TABLE IF EXISTS "' || table_schema || '".' || table_name;
-    EXECUTE 'DROP SCHEMA IF EXISTS ' || table_schema || ' CASCADE';
-    EXECUTE 'DROP SERVER IF EXISTS ' || servername || ' CASCADE;';
-    RETURN true;
-END;
-$$ LANGUAGE plpgsql SECURITY DEFINER;

src/pg/test/expected/01_install_test.out

@@ -1,5 +1,4 @@
 -- Install dependencies
 CREATE EXTENSION postgis;
-CREATE EXTENSION postgres_fdw;
 -- Install the extension
 CREATE EXTENSION observatory VERSION 'dev';

src/pg/test/expected/41_observatory_augmentation_test.out

@@ -3,36 +3,12 @@
 obs_getdemographicsnapshot_test_no_returns
 t
 (1 row)
-obs_get_median_income_at_test_point
-t
-(1 row)
-obs_get_median_income_at_null_island
-t
-(1 row)
-obs_getpoints_for_test_point_value|obs_getpoints_for_test_point_name|obs_getpoints_for_test_point_tablename|obs_getpoints_for_test_point_aggregate|obs_getpoints_for_test_point_type|obs_getpoints_for_test_point_description
-t|t|t|t|t|t
-(1 row)
-obs_getpoints_for_null_island
-t
-(1 row)
-obs_getpolygons_for_test_point
-t
-(1 row)
-obs_getpolygons_for_null_island
-t
-(1 row)
 test_point_segmentation
 t
 (1 row)
 null_island_segmentation
 t
 (1 row)
-getcategories_at_test_point_1
-t
-(1 row)
-getcategories_at_null_island
-t
-(1 row)
 obs_getmeasure_zhvi_point_test
 t
 (1 row)
@@ -66,12 +42,30 @@ t
 obs_getmeasure_bad_geometry
 t
 (1 row)
+obs_getmeasure_null_geometry
+t
+(1 row)
+obs_getmeasure_out_of_bounds_geometry
+t
+(1 row)
+obs_getmeasure_estimate_for_blank_aggregate
+t
+(1 row)
+obs_getmeasure_per_capita_income_average
+t
+(1 row)
+obs_getmeasure_median_capita_income_average
+t
+(1 row)
 obs_getcategory_point
 t
 (1 row)
 obs_getcategory_polygon
 t
 (1 row)
+obs_getcategory_null
+t
+(1 row)
 obs_getpopulation
 t
 (1 row)
@@ -81,6 +75,9 @@ t
 obs_getpopulation_polygon_null_test
 t
 (1 row)
+obs_getpopulation_polygon_null_geom_test
+t
+(1 row)
 obs_getuscensusmeasure_point_male_pop
 t
 (1 row)
@@ -90,12 +87,18 @@ t
 obs_getuscensusmeasure_null
 t
 (1 row)
+obs_getuscensusmeasure_null_geom
+t
+(1 row)
 obs_getuscensuscategory_point
 t
 (1 row)
 obs_getuscensuscategory_polygon
 t
 (1 row)
+obs_getuscensuscategory_null
+t
+(1 row)
 obs_getmeasurebyid_cartodb_census_tract
 t
 (1 row)
@@ -108,3 +111,147 @@ t
 obs_getmeasurebyid_nulls
 t
 (1 row)
+obs_getmeasurebyid_null_id
+t
+(1 row)
+obs_getmeta_null_null_is_null
+t
+(1 row)
+obs_getmeta_null_empty_is_null
+t
+(1 row)
+obs_getmeta_nullisland_null_is_null
+t
+(1 row)
+obs_getmeta_nullisland_empty_is_null
+t
+(1 row)
+obs_getmeta_nullisland_us_measure_is_null
+t
+(1 row)
+id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|geom_id|normalization
+t|t|t|t|t|t|t|t|t|t|t
+(1 row)
+id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|denom_aggregate|denom_colname|denom_type|denom_name|geom_id|normalization
+t|t|t|t|t|t|t|t|t|t|t|t|t|t|t
+(1 row)
+id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|geom_id|normalization
+t|t|t|t|t|t|t|t|t|t|t
+(1 row)
+id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|denom_aggregate|denom_colname|denom_type|denom_name|geom_id|normalization
+t|t|t|t|t|t|t|t|t|t|t|t|t|t|t
+(1 row)
+id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|denom_aggregate|denom_colname|denom_type|denom_name|geom_id|normalization|id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|denom_aggregate|denom_colname|denom_type|denom_name|geom_id|normalization
+t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t
+(1 row)
+id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|denom_aggregate|denom_colname|denom_type|denom_name|geom_id|normalization
+t|t|t|t|t|t|t|t|t|t|t|t|t|t|t
+(1 row)
+obs_getmeta_conflicting_metadata
+t
+(1 row)
+obs_getdata_geomval_empty_null
+t
+(1 row)
+obs_getdata_text_empty_null
+t
+(1 row)
+obs_getdata_geomval_empty_one_measure
+t
+(1 row)
+id|data_point_measure_null|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_null|nullcol
+t|t|t
+(1 row)
+id|data_point_measure_area|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_area|nullcol
+t|t|t
+(1 row)
+id|data_point_measure_predenominated|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_predenominated|nullcol
+t|t|t
+(1 row)
+id|data_point_measure_impossible_denominated|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_impossible_denominated|nullcol
+t|t|t
+(1 row)
+id|data_point_measure_denominated|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_denominated|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_one_null|data_polygon_measure_two_null
+t|t|t
+(1 row)
+id|data_polygon_measure_one_predenom|data_polygon_measure_two_predenom
+t|t|t
+(1 row)
+id|data_polygon_measure_one_area|data_polygon_measure_two_area
+t|t|t
+(1 row)
+id|data_polygon_measure_tract|data_polygon_measure_bg
+t|t|t
+(1 row)
+id|data_point_categorical|nullcol
+t|t|t
+(1 row)
+id|data_poly_categorical|nullcol
+t|t|t
+(1 row)
+id|data_poly_categorical|valcol
+t|t|t
+(1 row)
+id|correct_num_geoms
+t|t
+(1 row)
+id|correct_num_geoms|correct_pop
+t|t|t
+(1 row)
+id|correct_num_geoms|correct_pop|correct_bg_names
+t|t|t|t
+(1 row)
+id|correct_num_points
+t|t
+(1 row)
+id|correct_num_points|pointgeom_names
+t|t|t
+(1 row)
+id|obs_getdata_by_id_one_measure_null
+t|t
+(1 row)
+id|obs_getdata_by_id_one_measure_predenom
+t|t
+(1 row)
+id|obs_getdata_by_id_one_measure_null|obs_getdata_by_id_two_measure_null
+t|t|t
+(1 row)
+id|obs_getdata_by_id_categorical
+t|t
+(1 row)
+id|obs_getdata_by_id_geometry
+t|t
+(1 row)
+obs_getdata_api_geomvals_no_args
+t
+(1 row)
+obs_getdata_api_geomvals_args_numer_return
+t
+(1 row)
+obs_getdata_api_geomvals_args_string_return
+t
+(1 row)
+obs_getdata_api_geomrefs_args_numer_return
+t
+(1 row)
+obs_getdata_api_geomrefs_args_string_return
+t
+(1 row)

src/pg/test/expected/42_observatory_exploration_test.out

@@ -12,3 +12,198 @@ t
 _obs_getavailableboundariesexist
 t
 (1 row)
+_obs_getavailablenumerators_usa_pop_in_all
+t
+(1 row)
+_obs_getavailablenumerators_usa_pop_in_nyc_point
+t
+(1 row)
+_obs_getavailablenumerators_usa_pop_in_usa_extents
+t
+(1 row)
+_obs_getavailablenumerators_no_usa_pop_not_in_zero_point
+t
+(1 row)
+_obs_getavailablenumerators_usa_pop_in_age_gender_subsection
+t
+(1 row)
+_obs_getavailablenumerators_no_pop_in_income_subsection
+t
+(1 row)
+_obs_getavailablenumerators_male_pop_denom_by_total_pop
+t
+(1 row)
+_obs_getavailablenumerators_no_income_denom_by_total_pop
+t
+(1 row)
+_obs_getavailablenumerators_zillow_at_zcta5
+t
+(1 row)
+_obs_getavailablenumerators_no_zillow_at_block_group
+t
+(1 row)
+_obs_getavailablenumerators_total_pop_2010_2014
+t
+(1 row)
+_obs_getavailablenumerators_no_total_pop_1996
+t
+(1 row)
+_obs_getavailabledenominators_usa_pop_in_all
+t
+(1 row)
+_obs_getavailabledenominators_usa_pop_in_nyc_point
+t
+(1 row)
+_obs_getavailabledenominators_usa_pop_in_usa_extents
+t
+(1 row)
+_obs_getavailabledenominators_no_usa_pop_not_in_zero_point
+t
+(1 row)
+_obs_getavailabledenominators_usa_pop_in_age_gender_subsection
+t
+(1 row)
+_obs_getavailabledenominators_no_pop_in_income_subsection
+t
+(1 row)
+_obs_getavailabledenominators_male_pop_denom_by_total_pop
+t
+(1 row)
+_obs_getavailabledenominators_no_income_denom_by_total_pop
+t
+(1 row)
+_obs_getavailabledenominators_at_zcta5
+t
+(1 row)
+_obs_getavailabledenominators_none_spanish_geom
+t
+(1 row)
+_obs_getavailabledenominators_total_pop_2010_2014
+t
+(1 row)
+_obs_getavailabledenominators_no_total_pop_1996
+t
+(1 row)
+_obs_getavailablegeometries_usa_bg_in_all
+t
+(1 row)
+_obs_getavailablegeometries_usa_bg_in_nyc_point
+t
+(1 row)
+_obs_getavailablegeometries_usa_bg_in_usa_extents
+t
+(1 row)
+_obs_getavailablegeometries_no_usa_bg_not_in_zero_point
+t
+(1 row)
+_obs_getavailablegeometries_usa_bg_in_boundary_subsection
+t
+(1 row)
+_obs_getavailablegeometries_no_bg_in_uk_section
+t
+(1 row)
+_obs_getavailablegeometries_total_pop_in_usa_bg
+t
+(1 row)
+_obs_getavailablegeometries_foobarbaz_not_in_usa_bg
+t
+(1 row)
+_obs_getavailablegeometries_total_pop_denom_in_usa_bg
+t
+(1 row)
+_obs_getavailablegeometries_foobarbaz_denom_not_in_usa_bg
+t
+(1 row)
+_obs_getavailablegeometries_bg_2015
+t
+(1 row)
+_obs_getavailablegeometries_bg_not_1996
+t
+(1 row)
+_obs_getavailabletimespans_2010_2014_in_all
+t
+(1 row)
+_obs_getavailabletimespans_2010_2014_in_nyc_point
+t
+(1 row)
+_obs_getavailabletimespans_2010_2014_in_usa_extents
+t
+(1 row)
+_obs_getavailabletimespans_no_usa_bg_not_in_zero_point
+t
+(1 row)
+_obs_getavailabletimespans_total_pop_in_2010_2014
+t
+(1 row)
+_obs_getavailabletimespans_foobarbaz_not_in_2010_2014
+t
+(1 row)
+_obs_getavailablegeometries_total_pop_denom_in_2010_2014
+t
+(1 row)
+_obs_getavailablegeometries_foobarbaz_denom_not_in_2010_2014
+t
+(1 row)
+_obs_geometryscores_500m_buffer
+t
+(1 row)
+_obs_geometryscores_5km_buffer
+t
+(1 row)
+_obs_geometryscores_50km_buffer
+t
+(1 row)
+_obs_geometryscores_500km_buffer
+t
+(1 row)
+_obs_geometryscores_2500km_buffer
+t
+(1 row)
+_obs_geometryscores_numgeoms_500m_buffer
+t
+(1 row)
+_obs_geometryscores_numgeoms_5km_buffer
+t
+(1 row)
+_obs_geometryscores_numgeoms_50km_buffer
+t
+(1 row)
+_obs_geometryscores_numgeoms_500km_buffer
+t
+(1 row)
+_obs_geometryscores_numgeoms_2500km_buffer
+t
+(1 row)
+_obs_geometryscores_500km_buffer_50_geoms
+t
+(1 row)
+_obs_geometryscores_500km_buffer_500_geoms
+t
+(1 row)
+_obs_geometryscores_500km_buffer_2500_geoms
+t
+(1 row)
+_obs_geometryscores_500km_buffer_25000_geoms
+t
+(1 row)
+_total_pop_in_legacy_builder_metadata
+t
+(1 row)
+_median_income_in_legacy_builder_metadata
+t
+(1 row)
+_gini_in_legacy_builder_metadata
+t
+(1 row)
+_total_pop_in_legacy_builder_metadata_sums
+t
+(1 row)
+_median_income_in_legacy_builder_metadata_sums
+t
+(1 row)
+_gini_not_in_legacy_builder_metadata_sums
+t
+(1 row)
+_no_dupe_subsections_in_legacy_builder_metadata
+t
+(1 row)

src/pg/test/expected/44_observatory_geometries_test.out

@@ -42,6 +42,12 @@ t
 obs_getboundarybyid_boundary_id_mismatch_geom_id
 t
 (1 row)
+_obs_getboundariesbygeometry_roads_around_cartodb
+t
+(1 row)
+_obs_getboundariesbygeometry_points_around_cartodb
+t
+(1 row)
 _obs_getboundariesbygeometry_tracts_around_cartodb
 t
 (1 row)
@@ -87,9 +93,3 @@ t
 obs_getpointsbypointandradius_around_null_island
 t
 (1 row)
-geoid_name_matches|table_name_matches|geom_name_matches
-t|t|t
-(1 row)
-geoid_name_matches|table_name_matches|geom_name_matches
-t|t|t
-(1 row)

src/pg/test/fixtures/drop_fixtures.sql

@@ -7,18 +7,27 @@ DROP TABLE IF EXISTS observatory.obs_column_tag;
 DROP TABLE IF EXISTS observatory.obs_tag;
 DROP TABLE IF EXISTS observatory.obs_column_to_column;
 DROP TABLE IF EXISTS observatory.obs_dump_version;
+DROP TABLE IF EXISTS observatory.obs_meta;
+DROP TABLE IF EXISTS observatory.obs_meta_numer;
+DROP TABLE IF EXISTS observatory.obs_meta_denom;
+DROP TABLE IF EXISTS observatory.obs_meta_geom;
+DROP TABLE IF EXISTS observatory.obs_meta_timespan;
+DROP TABLE IF EXISTS observatory.obs_column_table_tile;
+DROP TABLE IF EXISTS observatory.obs_column_table_tile_simple;
+DROP TABLE IF EXISTS observatory.obs_78fb6c1d6ff6505225175922c2c389ce48d7632c;
 DROP TABLE IF EXISTS observatory.obs_65f29658e096ca1485bf683f65fdbc9f05ec3c5d;
 DROP TABLE IF EXISTS observatory.obs_1746e37b7cd28cb131971ea4187d42d71f09c5f3;
+DROP TABLE IF EXISTS observatory.obs_fcd4e4f5610f6764973ef8c0c215b2e80bec8963;
+DROP TABLE IF EXISTS observatory.obs_c4411eba732408d47d73281772dbf03d60645dec;
 DROP TABLE IF EXISTS observatory.obs_1a098da56badf5f32e336002b0a81708c40d29cd;
 DROP TABLE IF EXISTS observatory.obs_7615e8622a68bfc5fe37c69c9880edfb40250103;
-DROP TABLE IF EXISTS observatory.obs_1babf5a26a1ecda5fb74963e88408f71d0364b81;
-DROP TABLE IF EXISTS observatory.obs_8764a6b439a4f8714f54d4b3a157bc5e36519066;
-DROP TABLE IF EXISTS observatory.obs_b393b5b88c6adda634b2071a8005b03c551b609a;
-DROP TABLE IF EXISTS observatory.obs_1ea93bbc109c87c676b3270789dacf7a1430db6c;
-DROP TABLE IF EXISTS observatory.obs_fc050f0b8673cfe3c6aa1040f749eb40975691b7;
+DROP TABLE IF EXISTS observatory.obs_a01cd5d8ccaa6531cef715071e9307e6b1987ec3;
 DROP TABLE IF EXISTS observatory.obs_6c1309a64d8f3e6986061f4d1ca7b57743e75e74;
+DROP TABLE IF EXISTS observatory.obs_0310c639744a2014bb1af82709228f05b59e7d3d;
+DROP TABLE IF EXISTS observatory.obs_87a814e485deabe3b12545a537f693d16ca702c2;
+DROP TABLE IF EXISTS observatory.obs_e32f8e59c7c8861ee5ee4029b3ace2af9a5c9caf;
+DROP TABLE IF EXISTS observatory.obs_1ea93bbc109c87c676b3270789dacf7a1430db6c;
+DROP TABLE IF EXISTS observatory.obs_b393b5b88c6adda634b2071a8005b03c551b609a;
+DROP TABLE IF EXISTS observatory.obs_8e30e6b3792430b410ba5b9e49cdc6a0d404d48f;
+DROP TABLE IF EXISTS observatory.obs_08025e1287e3af2b5de571d06562ba8d3bdb48e9;
 DROP TABLE IF EXISTS observatory.obs_d39f7fe5959891c8296490d83c22ded31c54af13;
-DROP TABLE IF EXISTS observatory.obs_144e8b4f906885b2e057ac4842644a553ae49c6e;
-DROP TABLE IF EXISTS observatory.obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308;
-
-DROP TABLE IF EXISTS observatory.obs_meta;

src/pg/test/sql/01_install_test.sql

@@ -1,6 +1,5 @@
 -- Install dependencies
 CREATE EXTENSION postgis;
-CREATE EXTENSION postgres_fdw;
 
 -- Install the extension
 CREATE EXTENSION observatory VERSION 'dev';

src/pg/test/sql/40_observatory_utility_test.sql

@@ -11,8 +11,8 @@ SELECT
   cdb_observatory._OBS_GeomTable(
     ST_SetSRID(ST_Point(-74.0059, 40.7128), 4326),
     'us.census.tiger.census_tract',
-    '2014'
-  ) = 'obs_fc050f0b8673cfe3c6aa1040f749eb40975691b7' As _obs_geomtable_with_returned_table;
+    '2015'
+  ) = 'obs_87a814e485deabe3b12545a537f693d16ca702c2' As _obs_geomtable_with_returned_table;
 
 -- get null for unknown geometry_id
 -- should give back null

src/pg/test/sql/41_observatory_augmentation_test.sql

@@ -5,139 +5,32 @@ SET client_min_messages TO WARNING;
 --
 WITH result as(
   Select count(coalesce(OBS_GetDemographicSnapshot->>'value', 'foo')) expected_columns
-  FROM cdb_observatory.OBS_GetDemographicSnapshot(cdb_observatory._TestPoint())
+  FROM cdb_observatory.OBS_GetDemographicSnapshot(cdb_observatory._TestPoint(), '2010 - 2014')
 ) select expected_columns = 52 as OBS_GetDemographicSnapshot_test_no_returns
 FROM result;
 
-WITH result as (
-  SELECT _OBS_Get::text as expected FROM
-  cdb_observatory._OBS_Get(
-    cdb_observatory._TestPoint(),
-    Array['us.census.acs.B19013001']::text[],
-    '2010 - 2014',
-    'us.census.tiger.block_group'
-  )
-) SELECT expected = '{"value":79292,"name":"Median Household Income in the past 12 Months","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"median","type":"Numeric","description":"Within a geographic area, the median income received by every household on a regular basis before payments for personal income taxes, social security, union dues, medicare deductions, etc.  It includes income received from wages, salary, commissions, bonuses, and tips; self-employment income from own nonfarm or farm businesses, including proprietorships and partnerships; interest, dividends, net rental income, royalty income, or income from estates and trusts; Social Security or Railroad Retirement income; Supplemental Security Income (SSI); any cash public assistance or welfare payments from the state or local welfare office; retirement, survivor, or disability benefits; and any other sources of income received regularly such as Veterans'' (VA) payments, unemployment and/or worker''s compensation, child support, and alimony."}'
-  As OBS_Get_median_income_at_test_point
-  FROM result;
-
--- median income at null island
-WITH result as (
-  SELECT count(_OBS_Get) as expected FROM
-  cdb_observatory._OBS_Get(
-    ST_SetSRID(ST_Point(0, 0), 4326),
-    Array['us.census.acs.B19013001']::text[],
-    '2010 - 2014',
-    'us.census.tiger.block_group'
-  )
-) select expected = 0 as OBS_Get_median_income_at_null_island
-  from result;
-
--- OBS_GetPoints
--- obs_getpoints
--- --------------------
--- {4809.33511352425}
-
--- SELECT
---   (cdb_observatory._OBS_GetPoints(
---     cdb_observatory._TestPoint(),
---     'obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308'::text, -- block groups (see _obs_geomtable)
---     (Array['{"colname":"total_pop","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"sum","name":"Total Population","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates."}'::json])
---   ))[1]::text = '{"value":10923.093200390833950,"name":"Total Population","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"sum","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates."}'
---   as OBS_GetPoints_for_test_point;
-WITH cte As (
-SELECT
-  (cdb_observatory._OBS_GetPoints(
-    cdb_observatory._TestPoint(),
-    'obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308'::text, -- block groups (see _obs_geomtable)
-    (Array['{"colname":"total_pop","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"sum","name":"Total Population","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates."}'::json])
-  ))[1]
-  as OBS_GetPoints_for_test_point)
-SELECT
-  (abs((OBS_GetPoints_for_test_point ->> 'value')::numeric - 10923.093200390833950) / 10923.093200390833950) < 0.001 As OBS_GetPoints_for_test_point_value,
-  (OBS_GetPoints_for_test_point ->> 'name') = 'Total Population' As OBS_GetPoints_for_test_point_name,
-  (OBS_GetPoints_for_test_point ->> 'tablename') = 'obs_1a098da56badf5f32e336002b0a81708c40d29cd' As OBS_GetPoints_for_test_point_tablename,
-  (OBS_GetPoints_for_test_point ->> 'aggregate') = 'sum' As OBS_GetPoints_for_test_point_aggregate,
-  (OBS_GetPoints_for_test_point ->> 'type') = 'Numeric' As OBS_GetPoints_for_test_point_type,
-  (OBS_GetPoints_for_test_point ->> 'description') = 'The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates.' As OBS_GetPoints_for_test_point_description
-FROM cte;
-
--- what happens at null island
-
-SELECT
-  (cdb_observatory._OBS_GetPoints(
-    ST_SetSRID(ST_Point(0, 0), 4326),
-    'obs_1a098da56badf5f32e336002b0a81708c40d29cd'::text, -- see example in obs_geomtable
-    (Array['{"colname":"total_pop","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"sum","name":"Total Population","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates."}'::json])
-  ))[1]::text  is null
-  as OBS_GetPoints_for_null_island;
-
--- OBS_GetPolygons
---   obs_getpolygons
--- --------------------
---  {12996.8172420752}
-
-SELECT
-  (cdb_observatory._OBS_GetPolygons(
-    cdb_observatory._TestArea(),
-    'obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308'::text, -- see example in obs_geomtable
-    Array['{"colname":"total_pop","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"sum","name":"Total Population","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates."}'::json]
-))[1]::text = '{"value":12327.3133495107,"name":"Total Population","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"sum","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates."}'
-  as OBS_GetPolygons_for_test_point;
-
--- see what happens around null island
-SELECT
-  ((cdb_observatory._OBS_GetPolygons(
-    ST_Buffer(ST_SetSRID(ST_Point(0, 0), 4326)::geography, 500)::geometry,
-    'obs_1a098da56badf5f32e336002b0a81708c40d29cd'::text, -- see example in obs_geomtable
-    Array['{"colname":"total_pop","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"sum","name":"Total Population","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates."}'::json])
-  )[1]->>'value') is null
-  as OBS_GetPolygons_for_null_island;
-
 SELECT cdb_observatory.OBS_GetSegmentSnapshot(
     cdb_observatory._TestPoint(),
     'us.census.tiger.census_tract'
-)::text =
- '{"x10_segment":"Wealthy, urban without Kids","x55_segment":"Wealthy transplants displacing long-term local residents","us.census.acs.B01003001_quantile":"0.3235","us.census.acs.B01001002_quantile":"0.494716216216216","us.census.acs.B01001026_quantile":"0.183756756756757","us.census.acs.B01002001_quantile":"0.0752837837837838","us.census.acs.B03002003_quantile":"0.293162162162162","us.census.acs.B03002004_quantile":"0.455527027027027","us.census.acs.B03002006_quantile":"0.656405405405405","us.census.acs.B03002012_quantile":"0.840081081081081","us.census.acs.B05001006_quantile":"0.727135135135135","us.census.acs.B08006001_quantile":"0.688635135135135","us.census.acs.B08006002_quantile":"0.0204459459459459","us.census.acs.B08006008_quantile":"0.679324324324324","us.census.acs.B08006009_quantile":"0.996716216216216","us.census.acs.B08006011_quantile":"0.967418918918919","us.census.acs.B08006015_quantile":"0.512945945945946","us.census.acs.B08006017_quantile":"0.0504864864864865","us.census.acs.B09001001_quantile":"0.192405405405405","us.census.acs.B11001001_quantile":"0.331702702702703","us.census.acs.B14001001_quantile":"0.296283783783784","us.census.acs.B14001002_quantile":"0.045472972972973","us.census.acs.B14001005_quantile":"0.0442702702702703","us.census.acs.B14001006_quantile":"0.0829054054054054","us.census.acs.B14001007_quantile":"0.701135135135135","us.census.acs.B14001008_quantile":"0.404527027027027","us.census.acs.B15003001_quantile":"0.191824324324324","us.census.acs.B15003017_quantile":"0.864162162162162","us.census.acs.B15003022_quantile":"0.754297297297297","us.census.acs.B15003023_quantile":"0.350054054054054","us.census.acs.B16001001_quantile":"0.217635135135135","us.census.acs.B16001002_quantile":"0.85972972972973","us.census.acs.B16001003_quantile":"0.342851351351351","us.census.acs.B17001001_quantile":"0.51204054054054","us.census.acs.B17001002_quantile":"0.813540540540541","us.census.acs.B19013001_quantile":"0.0948648648648649","us.census.acs.B19083001_quantile":"0.678351351351351","us.census.acs.B19301001_quantile":"0.146108108108108","us.census.acs.B25001001_quantile":"0.149067567567568","us.census.acs.B25002003_quantile":"0","us.census.acs.B25004002_quantile":"0","us.census.acs.B25004004_quantile":"0.944554054054054","us.census.acs.B25058001_quantile":"0.398040540540541","us.census.acs.B25071001_quantile":"0.0596081081081081","us.census.acs.B25075001_quantile":"0","us.census.acs.B25075025_quantile":null}' as test_point_segmentation;
+)::JSONB =
+ '{"x10_segment": "Wealthy, urban without Kids", "x55_segment": "Wealthy transplants displacing long-term local residents", "us.census.acs.B01001002_quantile": "0.494716216216216", "us.census.acs.B01001026_quantile": "0.183756756756757", "us.census.acs.B01002001_quantile": "0.0752837837837838", "us.census.acs.B01003001_quantile": "0.3235", "us.census.acs.B03002003_quantile": "0.293162162162162", "us.census.acs.B03002004_quantile": "0.455527027027027", "us.census.acs.B03002006_quantile": "0.656405405405405", "us.census.acs.B03002012_quantile": "0.840081081081081", "us.census.acs.B05001006_quantile": "0.727135135135135", "us.census.acs.B08006001_quantile": "0.688635135135135", "us.census.acs.B08006002_quantile": "0.0204459459459459", "us.census.acs.B08006009_quantile": "0.679324324324324", "us.census.acs.B08006011_quantile": "0.996716216216216", "us.census.acs.B08006015_quantile": "0.967418918918919", "us.census.acs.B08006017_quantile": "0.512945945945946", "us.census.acs.B08301010_quantile": "0.994743243243243", "us.census.acs.B09001001_quantile": "0.0504864864864865", "us.census.acs.B11001001_quantile": "0.192405405405405", "us.census.acs.B14001001_quantile": "0.331702702702703", "us.census.acs.B14001002_quantile": "0.296283783783784", "us.census.acs.B14001005_quantile": "0.045472972972973", "us.census.acs.B14001006_quantile": "0.0442702702702703", "us.census.acs.B14001007_quantile": "0.0829054054054054", "us.census.acs.B14001008_quantile": "0.701135135135135", "us.census.acs.B15003001_quantile": "0.404527027027027", "us.census.acs.B15003017_quantile": "0.191824324324324", "us.census.acs.B15003022_quantile": "0.864162162162162", "us.census.acs.B15003023_quantile": "0.754297297297297", "us.census.acs.B16001001_quantile": "0.350054054054054", "us.census.acs.B16001002_quantile": "0.217635135135135", "us.census.acs.B16001003_quantile": "0.85972972972973", "us.census.acs.B17001001_quantile": "0.342851351351351", "us.census.acs.B17001002_quantile": "0.51204054054054", "us.census.acs.B19013001_quantile": "0.813540540540541", "us.census.acs.B19083001_quantile": "0.0948648648648649", "us.census.acs.B19301001_quantile": "0.678351351351351", "us.census.acs.B25001001_quantile": "0.146108108108108", "us.census.acs.B25002003_quantile": "0.149067567567568", "us.census.acs.B25004002_quantile": "0", "us.census.acs.B25004004_quantile": "0", "us.census.acs.B25058001_quantile": "0.944554054054054", "us.census.acs.B25071001_quantile": "0.398040540540541", "us.census.acs.B25075001_quantile": "0.0596081081081081", "us.census.acs.B25075025_quantile": "0"}'::JSONB as test_point_segmentation;
 
 -- segmentation around null island
 SELECT cdb_observatory.OBS_GetSegmentSnapshot(
     ST_SetSRID(ST_Point(0, 0), 4326),
     'us.census.tiger.census_tract'
-)::text = '{"x10_segment":null,"x55_segment":null,"us.census.acs.B01003001_quantile":null,"us.census.acs.B01001002_quantile":null,"us.census.acs.B01001026_quantile":null,"us.census.acs.B01002001_quantile":null,"us.census.acs.B03002003_quantile":null,"us.census.acs.B03002004_quantile":null,"us.census.acs.B03002006_quantile":null,"us.census.acs.B03002012_quantile":null,"us.census.acs.B05001006_quantile":null,"us.census.acs.B08006001_quantile":null,"us.census.acs.B08006002_quantile":null,"us.census.acs.B08006008_quantile":null,"us.census.acs.B08006009_quantile":null,"us.census.acs.B08006011_quantile":null,"us.census.acs.B08006015_quantile":null,"us.census.acs.B08006017_quantile":null,"us.census.acs.B09001001_quantile":null,"us.census.acs.B11001001_quantile":null,"us.census.acs.B14001001_quantile":null,"us.census.acs.B14001002_quantile":null,"us.census.acs.B14001005_quantile":null,"us.census.acs.B14001006_quantile":null,"us.census.acs.B14001007_quantile":null,"us.census.acs.B14001008_quantile":null,"us.census.acs.B15003001_quantile":null,"us.census.acs.B15003017_quantile":null,"us.census.acs.B15003022_quantile":null,"us.census.acs.B15003023_quantile":null,"us.census.acs.B16001001_quantile":null,"us.census.acs.B16001002_quantile":null,"us.census.acs.B16001003_quantile":null,"us.census.acs.B17001001_quantile":null,"us.census.acs.B17001002_quantile":null,"us.census.acs.B19013001_quantile":null,"us.census.acs.B19083001_quantile":null,"us.census.acs.B19301001_quantile":null,"us.census.acs.B25001001_quantile":null,"us.census.acs.B25002003_quantile":null,"us.census.acs.B25004002_quantile":null,"us.census.acs.B25004004_quantile":null,"us.census.acs.B25058001_quantile":null,"us.census.acs.B25071001_quantile":null,"us.census.acs.B25075001_quantile":null,"us.census.acs.B25075025_quantile":null}' as null_island_segmentation;
-
-WITH result as (
-  SELECT array_agg(_OBS_GetCategories) as expected FROM
-    cdb_observatory._OBS_GetCategories(
-      cdb_observatory._TestPoint(),
-      Array['us.census.spielman_singleton_segments.X10'],
-      'us.census.tiger.census_tract'
-  )
-)
-  select (expected)[1]::text = '{"category":"Wealthy, urban without Kids","name":"Spielman-Singleton Segments: 10 Clusters","tablename":"obs_65f29658e096ca1485bf683f65fdbc9f05ec3c5d","aggregate":null,"type":"Text","description":"Sociodemographic classes from Spielman and Singleton 2015, 10 clusters"}' as GetCategories_at_test_point_1
-  from result;
-
-WITH result as (
-  SELECT array_agg(_OBS_GetCategories) as expected FROM
-    cdb_observatory._OBS_GetCategories(
-      ST_SetSRID(ST_Point(0,0), 4326),
-      Array['us.census.spielman_singleton_segments.X10'],
-      'us.census.tiger.census_tract'
-  )
-)
-  select expected[0] is NULL as GetCategories_at_null_island
-  from result;
+)::text is null as null_island_segmentation;
 
 -- Point-based OBS_GetMeasure with zillow
-SELECT abs(OBS_GetMeasure_zhvi_point - 583600) / 583600 < 0.001 AS OBS_GetMeasure_zhvi_point_test FROM cdb_observatory.OBS_GetMeasure(
+SELECT abs(OBS_GetMeasure_zhvi_point - 597900) / 597900 < 5.0 AS OBS_GetMeasure_zhvi_point_test FROM cdb_observatory.OBS_GetMeasure(
   ST_SetSRID(ST_Point(-73.94602417945862, 40.6768220087458), 4326),
   'us.zillow.AllHomes_Zhvi', null, 'us.census.tiger.zcta5', '2014-01'
 ) As t(OBS_GetMeasure_zhvi_point);
 
--- Point-based OBS_GetMeasure with zillow default to latest
-SELECT abs(OBS_GetMeasure_zhvi_point_default_latest - 972900) / 972900 < 0.001 AS OBS_GetMeasure_zhvi_point_default_latest_test FROM cdb_observatory.OBS_GetMeasure(
+-- Point-based OBS_GetMeasure with later measure
+SELECT abs(OBS_GetMeasure_zhvi_point_default_latest - 995400) / 995400 < 5.0 AS OBS_GetMeasure_zhvi_point_default_latest_test FROM cdb_observatory.OBS_GetMeasure(
   ST_SetSRID(ST_Point(-73.94602417945862, 40.6768220087458), 4326),
-  'us.zillow.AllHomes_Zhvi'
+  'us.zillow.AllHomes_Zhvi', null, 'us.census.tiger.zcta5', '2016-06'
 ) As t(OBS_GetMeasure_zhvi_point_default_latest);
 
 -- Point-based OBS_GetMeasure, default normalization (area)
@@ -196,20 +89,49 @@ SELECT (abs(cdb_observatory.OBS_GetMeasure(
 -- Poly-based OBS_GetMeasure with denominator normalization
 SELECT abs(cdb_observatory.OBS_GetMeasure(
   cdb_observatory._TestArea(),
-  'us.census.acs.B01001002', 'denominator') - 0.49026340444793965457) / 0.49026340444793965457 < 0.001 As OBS_GetMeasure_total_male_poly_denominator;
+  'us.census.acs.B01001002', 'denominator', null, '2010 - 2014') - 0.49026340444793965457) / 0.49026340444793965457 < 0.001 As OBS_GetMeasure_total_male_poly_denominator;
 
 -- Poly-based OBS_GetMeasure with one very bad geom
 SELECT abs(cdb_observatory.OBS_GetMeasure(
   cdb_observatory._ProblemTestArea(),
   'us.census.acs.B01003001') - 96230.2929825897) / 96230.2929825897 < 0.001 As OBS_GetMeasure_bad_geometry;
 
+-- OBS_GetMeasure with NULL Input geometry
+SELECT cdb_observatory.OBS_GetMeasure(
+  NULL,
+  'us.census.acs.B01003001') IS NULL As OBS_GetMeasure_null_geometry;
+
+-- OBS_GetMeasure where there is no data
+SELECT cdb_observatory.OBS_GetMeasure(
+  ST_SetSRID(st_point(0, 0), 4326),
+  'us.census.acs.B01003001') IS NULL As OBS_GetMeasure_out_of_bounds_geometry;
+
+-- OBS_GetMeasure over arbitrary area for a measure we cannot estimate
+SELECT cdb_observatory.OBS_GetMeasure(
+  ST_Buffer(cdb_observatory._testpoint(), 0.1),
+  'us.census.acs.B19083001') IS NULL As OBS_GetMeasure_estimate_for_blank_aggregate;
+
+-- OBS_GetMeasure over arbitrary area for an average measure we can estimate
+SELECT abs(cdb_observatory.OBS_GetMeasure(
+  ST_Buffer(cdb_observatory._testpoint(), 0.01),
+  'us.census.acs.B19301001') - 20025) / 20025 < 0.001 As OBS_GetMeasure_per_capita_income_average;
+
+-- OBS_GetMeasure over arbitrary area for a median measure we can estimate
+SELECT abs(cdb_observatory.OBS_GetMeasure(
+  ST_Buffer(cdb_observatory._testpoint(), 0.01),
+  'us.census.acs.B19013001') - 39266) / 39266 < 0.001 As OBS_GetMeasure_median_capita_income_average;
+
 -- Point-based OBS_GetCategory
 SELECT cdb_observatory.OBS_GetCategory(
   cdb_observatory._TestPoint(), 'us.census.spielman_singleton_segments.X10') = 'Wealthy, urban without Kids' As OBS_GetCategory_point;
 
 -- Poly-based OBS_GetCategory
 SELECT cdb_observatory.OBS_GetCategory(
-  cdb_observatory._TestArea(), 'us.census.spielman_singleton_segments.X10') = 'Wealthy, urban without Kids' As obs_getcategory_polygon;
+  cdb_observatory._TestArea(), 'us.census.spielman_singleton_segments.X10') = 'Hispanic and Young' As obs_getcategory_polygon;
+
+-- NULL Input OBS_GetCategory
+SELECT cdb_observatory.OBS_GetCategory(
+  NULL, 'us.census.spielman_singleton_segments.X10') IS NULL As obs_getcategory_null;
 
 -- Point-based OBS_GetPopulation, default normalization (area)
 SELECT (abs(OBS_GetPopulation - 10923.093200390833950) / 10923.093200390833950) < 0.001 As OBS_GetPopulation FROM
@@ -231,6 +153,13 @@ FROM
     cdb_observatory._TestArea(), NULL
   ) As m(obs_getpopulation_polygon_null);
 
+-- Null input OBS_GetPopulation
+SELECT obs_getpopulation_polygon_null_geom IS NULL As obs_getpopulation_polygon_null_geom_test
+FROM
+  cdb_observatory.OBS_GetPopulation(
+    NULL, NULL
+  ) As m(obs_getpopulation_polygon_null_geom);
+
 -- Point-based OBS_GetUSCensusMeasure, default normalization (area)
 SELECT (abs(cdb_observatory.obs_getuscensusmeasure(
   cdb_observatory._testpoint(), 'male population') - 6789.5647735060920500) / 6789.5647735060920500) < 0.001 As obs_getuscensusmeasure_point_male_pop;
@@ -244,13 +173,22 @@ SELECT (abs(cdb_observatory.obs_getuscensusmeasure(
 SELECT (abs(cdb_observatory.obs_getuscensusmeasure(
   cdb_observatory._testarea(), 'male population', NULL) - 6043.63061042765) / 6043.63061042765) < 0.001 As obs_getuscensusmeasure_null;
 
+-- Poly-based OBS_GetUSCensusMeasure, Null input geom
+SELECT cdb_observatory.obs_getuscensusmeasure(
+  NULL, 'male population', NULL) IS NULL As obs_getuscensusmeasure_null_geom;
+
+
 -- Point-based OBS_GetUSCensusCategory
 SELECT cdb_observatory.OBS_GetUSCensusCategory(
   cdb_observatory._testpoint(), 'Spielman-Singleton Segments: 10 Clusters') = 'Wealthy, urban without Kids' As OBS_GetUSCensusCategory_point;
 
 -- Area-based OBS_GetUSCensusCategory
 SELECT cdb_observatory.OBS_GetUSCensusCategory(
-  cdb_observatory._testarea(), 'Spielman-Singleton Segments: 10 Clusters') = 'Wealthy, urban without Kids' As OBS_GetUSCensusCategory_polygon;
+  cdb_observatory._testarea(), 'Spielman-Singleton Segments: 10 Clusters') = 'Hispanic and Young' As OBS_GetUSCensusCategory_polygon;
+
+-- Null-input OBS_GetUSCensusCategory
+SELECT cdb_observatory.OBS_GetUSCensusCategory(
+  NULL, 'Spielman-Singleton Segments: 10 Clusters') IS NULL As OBS_GetUSCensusCategory_null;
 
 
 -- OBS_GetMeasureById tests
@@ -285,3 +223,543 @@ SELECT cdb_observatory.OBS_GetMeasureById(
   'us.census.tiger.block_group',
   '2010 - 2014'
 ) IS NULL As OBS_GetMeasureById_nulls;
+
+-- NULL input id
+SELECT cdb_observatory.OBS_GetMeasureById(
+  NULL,
+  'us.census.acs.B01003001',
+  'us.census.tiger.block_group',
+  '2010 - 2014'
+) IS NULL As OBS_GetMeasureById_null_id;
+
+-- OBS_GetMeta null/null
+SELECT cdb_observatory.OBS_GetMeta(NULL, NULL) IS NULL
+AS OBS_GetMeta_null_null_is_null;
+
+-- OBS_GetMeta null/empty array
+SELECT cdb_observatory.OBS_GetMeta(NULL, '[]') IS NULL
+AS OBS_GetMeta_null_empty_is_null;
+
+-- OBS_GetMeta nullisland/null
+SELECT cdb_observatory.OBS_GetMeta(ST_Point(0, 0), NULL) IS NULL
+AS OBS_GetMeta_nullisland_null_is_null;
+
+-- OBS_GetMeta nullisland/empty array
+SELECT cdb_observatory.OBS_GetMeta(ST_Point(0, 0), '[]') IS NULL
+AS OBS_GetMeta_nullisland_empty_is_null;
+
+-- OBS_GetMeta nullisland/us_measure data
+SELECT cdb_observatory.OBS_GetMeta(ST_Point(0, 0),
+  '[{"numer_id": "us.census.acs.B01003001"}]') IS NULL
+AS OBS_GetMeta_nullisland_us_measure_is_null;
+
+-- OBS_GetMeta for point completes one partial measure with "best" metadata
+-- with no denominator
+WITH meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01003001"}]') meta)
+SELECT
+(meta->0->>'id')::integer = 1 id,
+(meta->0->>'numer_id') = 'us.census.acs.B01003001' numer_id,
+(meta->0->>'timespan_rank')::integer = 1 timespan_rank,
+(meta->0->>'score_rank')::integer = 1 score_rank,
+(meta->0->>'numer_aggregate') = 'sum' numer_aggregate,
+(meta->0->>'numer_colname') = 'total_pop' numer_colname,
+(meta->0->>'numer_type') = 'Numeric' numer_type,
+(meta->0->>'numer_name') = 'Total Population' numer_name,
+(meta->0->>'denom_id') IS NULL denom_id,
+(meta->0->>'geom_id') = 'us.census.tiger.block_group' geom_id,
+(meta->0->>'normalization') IS NULL normalization
+FROM meta;
+
+-- OBS_GetMeta for point completes one partial measure with "best" metadata
+-- with a denominator
+WITH meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01001002"}]') meta)
+SELECT
+(meta->0->>'id')::integer = 1 id,
+(meta->0->>'numer_id') = 'us.census.acs.B01001002' numer_id,
+(meta->0->>'timespan_rank')::integer = 1 timespan_rank,
+(meta->0->>'score_rank')::integer = 1 score_rank,
+(meta->0->>'numer_aggregate') = 'sum' numer_aggregate,
+(meta->0->>'numer_colname') = 'male_pop' numer_colname,
+(meta->0->>'numer_type') = 'Numeric' numer_type,
+(meta->0->>'numer_name') = 'Male Population' numer_name,
+(meta->0->>'denom_id') = 'us.census.acs.B01003001' denom_id,
+(meta->0->>'denom_aggregate') = 'sum' denom_aggregate,
+(meta->0->>'denom_colname') = 'total_pop' denom_colname,
+(meta->0->>'denom_type') = 'Numeric' denom_type,
+(meta->0->>'denom_name') = 'Total Population' denom_name,
+(meta->0->>'geom_id') = 'us.census.tiger.block_group' geom_id,
+(meta->0->>'normalization') IS NULL normalization
+FROM meta;
+
+-- OBS_GetMeta for polygon completes one partial measure with "best" metadata
+-- with no denominator
+WITH meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001"}]') meta)
+SELECT
+(meta->0->>'id')::integer = 1 id,
+(meta->0->>'numer_id') = 'us.census.acs.B01003001' numer_id,
+(meta->0->>'timespan_rank')::integer = 1 timespan_rank,
+(meta->0->>'score_rank')::integer = 1 score_rank,
+(meta->0->>'numer_aggregate') = 'sum' numer_aggregate,
+(meta->0->>'numer_colname') = 'total_pop' numer_colname,
+(meta->0->>'numer_type') = 'Numeric' numer_type,
+(meta->0->>'numer_name') = 'Total Population' numer_name,
+(meta->0->>'denom_id') IS NULL denom_id,
+(meta->0->>'geom_id') = 'us.census.tiger.block_group' geom_id,
+(meta->0->>'normalization') IS NULL normalization
+FROM meta;
+
+-- OBS_GetMeta for polygon completes one partial measure with "best" metadata
+-- with a denominator
+WITH meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01001002"}]') meta)
+SELECT
+(meta->0->>'id')::integer = 1 id,
+(meta->0->>'numer_id') = 'us.census.acs.B01001002' numer_id,
+(meta->0->>'timespan_rank')::integer = 1 timespan_rank,
+(meta->0->>'score_rank')::integer = 1 score_rank,
+(meta->0->>'numer_aggregate') = 'sum' numer_aggregate,
+(meta->0->>'numer_colname') = 'male_pop' numer_colname,
+(meta->0->>'numer_type') = 'Numeric' numer_type,
+(meta->0->>'numer_name') = 'Male Population' numer_name,
+(meta->0->>'denom_id') = 'us.census.acs.B01003001' denom_id,
+(meta->0->>'denom_aggregate') = 'sum' denom_aggregate,
+(meta->0->>'denom_colname') = 'total_pop' denom_colname,
+(meta->0->>'denom_type') = 'Numeric' denom_type,
+(meta->0->>'denom_name') = 'Total Population' denom_name,
+(meta->0->>'geom_id') = 'us.census.tiger.block_group' geom_id,
+(meta->0->>'normalization') IS NULL normalization
+FROM meta;
+
+-- OBS_GetMeta for point completes several partial measures with "best"
+-- metadata, includes geom alternatives if asked
+WITH meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01001002"}]', null, 2) meta)
+SELECT
+(meta->0->>'id')::integer = 1 id,
+(meta->0->>'numer_id') = 'us.census.acs.B01001002' numer_id,
+(meta->0->>'timespan_rank')::integer = 1 timespan_rank,
+(meta->0->>'score_rank')::integer = 1 score_rank,
+(meta->0->>'numer_aggregate') = 'sum' numer_aggregate,
+(meta->0->>'numer_colname') = 'male_pop' numer_colname,
+(meta->0->>'numer_type') = 'Numeric' numer_type,
+(meta->0->>'numer_name') = 'Male Population' numer_name,
+(meta->0->>'denom_id') = 'us.census.acs.B01003001' denom_id,
+(meta->0->>'denom_aggregate') = 'sum' denom_aggregate,
+(meta->0->>'denom_colname') = 'total_pop' denom_colname,
+(meta->0->>'denom_type') = 'Numeric' denom_type,
+(meta->0->>'denom_name') = 'Total Population' denom_name,
+(meta->0->>'geom_id') = 'us.census.tiger.block_group' geom_id,
+(meta->0->>'normalization') IS NULL normalization,
+(meta->1->>'id')::integer = 1 id,
+(meta->1->>'numer_id') = 'us.census.acs.B01001002' numer_id,
+(meta->1->>'timespan_rank')::integer = 1 timespan_rank,
+(meta->1->>'score_rank')::integer = 2 score_rank,
+(meta->1->>'numer_aggregate') = 'sum' numer_aggregate,
+(meta->1->>'numer_colname') = 'male_pop' numer_colname,
+(meta->1->>'numer_type') = 'Numeric' numer_type,
+(meta->1->>'numer_name') = 'Male Population' numer_name,
+(meta->1->>'denom_id') = 'us.census.acs.B01003001' denom_id,
+(meta->1->>'denom_aggregate') = 'sum' denom_aggregate,
+(meta->1->>'denom_colname') = 'total_pop' denom_colname,
+(meta->1->>'denom_type') = 'Numeric' denom_type,
+(meta->1->>'denom_name') = 'Total Population' denom_name,
+(meta->1->>'geom_id') = 'us.census.tiger.census_tract' geom_id,
+(meta->1->>'normalization') IS NULL normalization
+FROM meta;
+
+-- OBS_GetMeta for point completes several partial measures with "best" metadata
+-- with pre-computed geom
+WITH meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01001002", "geom_id": "us.census.tiger.census_tract"}]') meta)
+SELECT
+(meta->0->>'id')::integer = 1 id,
+(meta->0->>'numer_id') = 'us.census.acs.B01001002' numer_id,
+(meta->0->>'timespan_rank')::integer = 1 timespan_rank,
+(meta->0->>'score_rank')::integer = 1 score_rank,
+(meta->0->>'numer_aggregate') = 'sum' numer_aggregate,
+(meta->0->>'numer_colname') = 'male_pop' numer_colname,
+(meta->0->>'numer_type') = 'Numeric' numer_type,
+(meta->0->>'numer_name') = 'Male Population' numer_name,
+(meta->0->>'denom_id') = 'us.census.acs.B01003001' denom_id,
+(meta->0->>'denom_aggregate') = 'sum' denom_aggregate,
+(meta->0->>'denom_colname') = 'total_pop' denom_colname,
+(meta->0->>'denom_type') = 'Numeric' denom_type,
+(meta->0->>'denom_name') = 'Total Population' denom_name,
+(meta->0->>'geom_id') = 'us.census.tiger.census_tract' geom_id,
+(meta->0->>'normalization') IS NULL normalization
+FROM meta;
+
+-- OBS_GetMeta for point completes several partial measures with conflicting
+-- metadata
+SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01001002", "denom_id": "us.census.acs.B01001002", "geom_id": "us.census.tiger.census_tract"}]') IS NULL
+AS obs_getmeta_conflicting_metadata;
+
+-- OBS_GetData/OBS_GetMeta by id with empty list/null
+WITH data AS (SELECT * FROM cdb_observatory.OBS_GetData(ARRAY[]::TEXT[], null))
+SELECT ARRAY_AGG(data) IS NULL AS obs_getdata_geomval_empty_null FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with empty list/null
+WITH data AS (SELECT * FROM cdb_observatory.OBS_GetData(ARRAY[]::GEOMVAL[], null))
+SELECT ARRAY_AGG(data) IS NULL AS obs_getdata_text_empty_null FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with empty list
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01003001"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(ARRAY[]::GEOMVAL[],
+  (SELECT meta FROM meta)))
+SELECT ARRAY_AGG(data) IS NULL AS obs_getdata_geomval_empty_one_measure FROM data;
+
+-- OBS_GetData/OBS_GetMeta by point geom with one standard measure NULL
+-- normalization
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01003001"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestPoint(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 10923) / 10923 < 0.001 data_point_measure_null,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by polygon geom with one standard measure NULL
+-- normalization
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 15787) / 15787 < 0.001 data_polygon_measure_null,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by point geom with one standard measure area
+-- normalization
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "area"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestPoint(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 10923) / 10923 < 0.001 data_point_measure_area,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by polygon geom with one standard measure area
+-- normalization
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "area"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 15787) / 15787 < 0.001 data_polygon_measure_area,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by point geom with one standard measure predenom
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "predenominated"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestPoint(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 1900) / 1900 < 0.001 data_point_measure_predenominated,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by polygon geom with one standard measure predenom
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "predenominated"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 12327) / 12327 < 0.001 data_polygon_measure_predenominated,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by point geom with impossible denom
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "denominated"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestPoint(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       data->0->>'value' IS NULL data_point_measure_impossible_denominated,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by polygon geom with one impossible denom
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "denominated"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       data->0->>'value' IS NULL data_polygon_measure_impossible_denominated,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by point geom with denom
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01001002", "normalization": "denominated"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestPoint(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 0.6215) / 0.6215 < 0.001 data_point_measure_denominated,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by polygon geom with one denom measure
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01001002", "normalization": "denominated"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 0.4902) / 0.4902 < 0.001 data_polygon_measure_denominated,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with two standard measures NULL normalization
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001"}, {"numer_id": "us.census.acs.B01001002"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 15787) / 15787 < 0.001 data_polygon_measure_one_null,
+       abs((data->1->>'value')::Numeric - 0.4902) / 0.4902 < 0.001 data_polygon_measure_two_null
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with two standard measures predenom normalization
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "predenom"}, {"numer_id": "us.census.acs.B01001002", "normalization": "predenom"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 12327) / 12327 < 0.001 data_polygon_measure_one_predenom,
+       abs((data->1->>'value')::Numeric - 6043) / 6043 < 0.001 data_polygon_measure_two_predenom
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with two standard measures area normalization
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "area"}, {"numer_id": "us.census.acs.B01001002", "normalization": "area"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 15787) / 15787 < 0.001 data_polygon_measure_one_area,
+       abs((data->1->>'value')::Numeric - 7739) / 7739 < 0.001 data_polygon_measure_two_area
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with two standard measures different geoms
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.census_tract"}, {"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.block_group"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 16960) / 16960 < 0.001 data_polygon_measure_tract,
+       abs((data->1->>'value')::Numeric - 15787) / 15787 < 0.001 data_polygon_measure_bg
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by point geom with one categorical
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.spielman_singleton_segments.X55"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestPoint(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       data->0->>'value' = 'Wealthy transplants displacing long-term local residents' data_point_categorical,
+       data->1->>'value' IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by polygon geom with one categorical
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.spielman_singleton_segments.X55"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       data->0->>'value' = 'Hispanic Black mix multilingual, high poverty, renters, uses public transport' data_poly_categorical,
+       data->1->>'value' IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with one categorical and one measure
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.spielman_singleton_segments.X55"}, {"numer_id": "us.census.acs.B01003001"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       data->0->>'value' = 'Hispanic Black mix multilingual, high poverty, renters, uses public transport' data_poly_categorical,
+       abs((data->1->>'value')::Numeric - 15787) / 15787 < 0.0001 valcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with polygons inside a polygon
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.block_group"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta), false))
+SELECT every(id = 1) is TRUE id,
+       count(distinct (data->0->>'value')::geometry) = 16 correct_num_geoms
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with polygons inside a polygon + one measure
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.block_group"}, {"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.block_group"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta), false))
+SELECT every(id = 1) is TRUE id,
+       count(distinct (data->0->>'value')::geometry) = 16 correct_num_geoms,
+       abs(sum((data->1->>'value')::numeric) - 15787) / 15787 < 0.001 correct_pop
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with polygons inside a polygon + one measure + one text
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.block_group"}, {"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.block_group"}, {"numer_id": "us.census.tiger.name", "geom_id": "us.census.tiger.block_group"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta), false))
+SELECT every(id = 1) is TRUE id,
+       count(distinct (data->0->>'value')::geometry) = 16 correct_num_geoms,
+       abs(sum((data->1->>'value')::numeric) - 15787) / 15787 < 0.001 correct_pop,
+       array_agg(distinct data->2->>'value') = '{"Block Group 1","Block Group 2","Block Group 3","Block Group 4","Block Group 5"}' correct_bg_names
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with points inside a polygon
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.pointlm_geom"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta), false))
+SELECT every(id = 1) AS id,
+       count(distinct (data->0->>'value')::geometry(point, 4326)) = 3 correct_num_points
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with points inside a polygon + one text
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.pointlm_geom"}, {"geom_id": "us.census.tiger.pointlm_geom", "numer_id": "us.census.tiger.fullname"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta), false))
+SELECT every(id = 1) AS id,
+       count(distinct (data->0->>'value')::geometry(point, 4326)) = 3 correct_num_points,
+       array_agg(data->1->>'value') = '{"Bushwick Yards","Edward Block Square","Bushwick Houses"}' pointgeom_names
+FROM data;
+
+
+-- OBS_GetData by id with one standard measure
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.census_tract", "numer_id": "us.census.acs.B01003001"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY['36047048500'],
+    (SELECT meta FROM meta)))
+SELECT id = '36047048500' AS id,
+       (abs((data->0->>'value')::numeric) - 5578) / 5578 < 0.001 obs_getdata_by_id_one_measure_null
+FROM data;
+
+-- OBS_GetData by id with one standard measure, predenominated
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"normalization": "predenominated", "geom_id": "us.census.tiger.census_tract", "numer_id": "us.census.acs.B01003001"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY['36047048500'],
+    (SELECT meta FROM meta)))
+SELECT id = '36047048500' AS id,
+       (abs((data->0->>'value')::numeric) - 3241) / 3241 < 0.001 obs_getdata_by_id_one_measure_predenom
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by id with two standard measures
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.census_tract", "numer_id": "us.census.acs.B01003001"}, {"geom_id": "us.census.tiger.census_tract", "numer_id": "us.census.acs.B01001002"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY['36047048500'],
+    (SELECT meta FROM meta)))
+SELECT id = '36047048500' AS id,
+       (abs((data->0->>'value')::numeric) - 5578) / 5578 < 0.001 obs_getdata_by_id_one_measure_null,
+       (abs((data->1->>'value')::numeric) - 0.6053) / 0.6053 < 0.001 obs_getdata_by_id_two_measure_null
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by id with one categorical
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.census_tract", "numer_id": "us.census.spielman_singleton_segments.X55"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY['36047048500'],
+    (SELECT meta FROM meta)))
+SELECT id = '36047048500' AS id,
+       data->0->>'value' = 'Wealthy transplants displacing long-term local residents' obs_getdata_by_id_categorical
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by id with one geometry
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.census_tract"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY['36047048500'],
+    (SELECT meta FROM meta)))
+SELECT id = '36047048500' AS id,
+       ST_GeometryType((data->0->>'value')::geometry) = 'ST_MultiPolygon' obs_getdata_by_id_geometry
+FROM data;
+
+-- OBS_GetData with an API + geomvals, no args
+SELECT ARRAY['us.census.tiger.census_tract'] <@ array_agg(data->0->>'value') AS OBS_GetData_API_geomvals_no_args
+FROM cdb_observatory.obs_getdata(array[(cdb_observatory._testarea(), 1)::geomval],
+  '[{"numer_type": "text", "numer_colname": "boundary_id", "api_method": "obs_getavailableboundaries", "geom_geomref_colname": "boundary_id"}]',
+  false);
+
+-- OBS_GetData with an API + geomvals, args, numeric
+SELECT json_typeof(data->0->'value') = 'number' AS OBS_GetData_API_geomvals_args_numer_return
+FROM cdb_observatory.obs_getdata(array[(cdb_observatory._testarea(), 1)::geomval],
+    '[{"numer_type": "numeric", "numer_colname": "obs_getmeasure", "api_method": "obs_getmeasure", "api_args": ["us.census.acs.B01003001"]}]', false);
+
+-- OBS_GetData with an API + geomvals, args, text
+SELECT json_typeof(data->0->'value') = 'string' AS OBS_GetData_API_geomvals_args_string_return
+FROM cdb_observatory.obs_getdata(array[(cdb_observatory._testarea(), 1)::geomval],
+    '[{"numer_type": "text", "numer_colname": "obs_getcategory", "api_method": "obs_getcategory", "api_args": ["us.census.spielman_singleton_segments.X55"]}]', false);
+
+-- OBS_GetData with an API + geomrefs, args, numeric
+SELECT json_typeof(data->0->'value') = 'number' AS OBS_GetData_API_geomrefs_args_numer_return
+FROM cdb_observatory.obs_getdata(array['36047076200'],
+      '[{"numer_type": "numeric", "numer_colname": "obs_getmeasurebyid", "api_method": "obs_getmeasurebyid", "api_args": ["us.census.acs.B01003001", "us.census.tiger.census_tract"]}]');
+
+-- OBS_GetData with an API + geomrefs, args, text
+SELECT json_typeof(data->0->'value') = 'string' AS OBS_GetData_API_geomrefs_args_string_return
+FROM cdb_observatory.obs_getdata(array['36047'],
+      '[{"numer_type": "text", "numer_colname": "obs_getboundarybyid", "api_method": "obs_getboundarybyid", "api_args": ["us.census.tiger.county"]}]');

src/pg/test/sql/42_observatory_exploration_test.sql

@@ -10,11 +10,11 @@ SET client_min_messages TO WARNING;
 
 -- _OBS_SearchTables tests
 SELECT
-  t.table_name = 'obs_1babf5a26a1ecda5fb74963e88408f71d0364b81' As _OBS_SearchTables_tables_match,
-  t.timespan = '2014' As _OBS_SearchTables_timespan_matches
+  t.table_name = 'obs_0310c639744a2014bb1af82709228f05b59e7d3d' As _OBS_SearchTables_tables_match,
+  t.timespan = '2015' As _OBS_SearchTables_timespan_matches
 FROM cdb_observatory._OBS_SearchTables(
   'us.census.tiger.county',
-  '2014'
+  '2015'
 ) As t(table_name, timespan);
 
 -- _OBS_SearchTables tests
@@ -33,3 +33,497 @@ SELECT COUNT(*) > 0 AS _OBS_GetAvailableBoundariesExist
 FROM cdb_observatory.OBS_GetAvailableBoundaries(
   cdb_observatory._TestPoint()
 ) AS t(boundary_id, description, time_span, tablename);
+
+--
+-- OBS_GetAvailableNumerators tests
+--
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators())
+AS _obs_getavailablenumerators_usa_pop_in_all;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailablenumerators_usa_pop_in_nyc_point;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakeEnvelope(
+      -169.8046875, 21.289374355860424,
+      -47.4609375, 72.0739114882038
+  ), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailablenumerators_usa_pop_in_usa_extents;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(0, 0), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailablenumerators_no_usa_pop_not_in_zero_point;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  ARRAY['subsection/tags.age_gender']
+))
+AS _obs_getavailablenumerators_usa_pop_in_age_gender_subsection;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  ARRAY['subsection/tags.income']
+))
+AS _obs_getavailablenumerators_no_pop_in_income_subsection;
+
+SELECT 'us.census.acs.B01001002' IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, 'us.census.acs.B01003001'
+) WHERE valid_denom = True)
+AS _obs_getavailablenumerators_male_pop_denom_by_total_pop;
+
+SELECT 'us.census.acs.B19013001' NOT IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, 'us.census.acs.B01003001'
+) WHERE valid_denom = True)
+AS _obs_getavailablenumerators_no_income_denom_by_total_pop;
+
+SELECT 'us.zillow.AllHomes_Zhvi' IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, 'us.census.tiger.zcta5'
+) WHERE valid_geom = True)
+AS _obs_getavailablenumerators_zillow_at_zcta5;
+
+SELECT 'us.zillow.AllHomes_Zhvi' NOT IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, 'us.census.tiger.block_group'
+) WHERE valid_geom = True)
+AS _obs_getavailablenumerators_no_zillow_at_block_group;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, '2010 - 2014'
+) WHERE valid_timespan = True)
+AS _obs_getavailablenumerators_total_pop_2010_2014;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, '1996'
+) WHERE valid_timespan = True)
+AS _obs_getavailablenumerators_no_total_pop_1996;
+
+--
+-- OBS_GetAvailableDenominators tests
+--
+
+SELECT 'us.census.acs.B01003001' IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators())
+AS _obs_getavailabledenominators_usa_pop_in_all;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailabledenominators_usa_pop_in_nyc_point;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakeEnvelope(
+      -169.8046875, 21.289374355860424,
+      -47.4609375, 72.0739114882038
+  ), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailabledenominators_usa_pop_in_usa_extents;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(0, 0), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailabledenominators_no_usa_pop_not_in_zero_point;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  ARRAY['subsection/tags.age_gender']
+))
+AS _obs_getavailabledenominators_usa_pop_in_age_gender_subsection;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  ARRAY['subsection/tags.income']
+))
+AS _obs_getavailabledenominators_no_pop_in_income_subsection;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, 'us.census.acs.B01001002'
+) WHERE valid_numer = True)
+AS _obs_getavailabledenominators_male_pop_denom_by_total_pop;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, 'us.census.acs.B19013001'
+) WHERE valid_numer = True)
+AS _obs_getavailabledenominators_no_income_denom_by_total_pop;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, 'us.census.tiger.zcta5'
+) WHERE valid_geom = True)
+AS _obs_getavailabledenominators_at_zcta5;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, 'es.ine.the_geom'
+) WHERE valid_geom = True)
+AS _obs_getavailabledenominators_none_spanish_geom;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, '2010 - 2014'
+) WHERE valid_timespan = True)
+AS _obs_getavailabledenominators_total_pop_2010_2014;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, '1996'
+) WHERE valid_timespan = True)
+AS _obs_getavailabledenominators_no_total_pop_1996;
+
+--
+-- OBS_GetAvailableGeometries tests
+--
+
+SELECT 'us.census.tiger.block_group' IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries())
+AS _obs_getavailablegeometries_usa_bg_in_all;
+
+SELECT 'us.census.tiger.block_group' IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailablegeometries_usa_bg_in_nyc_point;
+
+SELECT 'us.census.tiger.block_group' IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakeEnvelope(
+      -169.8046875, 21.289374355860424,
+      -47.4609375, 72.0739114882038
+  ), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailablegeometries_usa_bg_in_usa_extents;
+
+SELECT 'us.census.tiger.block_group' NOT IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(0, 0), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailablegeometries_no_usa_bg_not_in_zero_point;
+
+SELECT 'us.census.tiger.block_group' IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  ARRAY['subsection/tags.boundary']
+))
+AS _obs_getavailablegeometries_usa_bg_in_boundary_subsection;
+
+SELECT 'us.census.tiger.block_group' NOT IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  ARRAY['section/tags.uk']
+))
+AS _obs_getavailablegeometries_no_bg_in_uk_section;
+
+SELECT 'us.census.tiger.block_group' IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, 'us.census.acs.B01003001'
+) WHERE valid_numer = True)
+AS _obs_getavailablegeometries_total_pop_in_usa_bg;
+
+SELECT 'us.census.tiger.block_group' NOT IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, 'foo.bar.baz'
+) WHERE valid_numer = True)
+AS _obs_getavailablegeometries_foobarbaz_not_in_usa_bg;
+
+SELECT 'us.census.tiger.block_group' IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, 'us.census.acs.B01003001'
+) WHERE valid_denom = True)
+AS _obs_getavailablegeometries_total_pop_denom_in_usa_bg;
+
+SELECT 'us.census.tiger.block_group' NOT IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, 'foo.bar.baz'
+) WHERE valid_denom = True)
+AS _obs_getavailablegeometries_foobarbaz_denom_not_in_usa_bg;
+
+SELECT 'us.census.tiger.block_group' IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, '2015'
+) WHERE valid_timespan = True)
+AS _obs_getavailablegeometries_bg_2015;
+
+SELECT 'us.census.tiger.block_group' NOT IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, '1996'
+) WHERE valid_timespan = True)
+AS _obs_getavailablegeometries_bg_not_1996;
+
+--
+-- OBS_GetAvailableTimespans tests
+--
+
+SELECT '2010 - 2014' IN (SELECT timespan_id
+FROM cdb_observatory.OBS_GetAvailableTimespans())
+AS _obs_getavailabletimespans_2010_2014_in_all;
+
+SELECT '2010 - 2014' IN (SELECT timespan_id
+FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailabletimespans_2010_2014_in_nyc_point;
+
+SELECT '2010 - 2014' IN (SELECT timespan_id
+FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_SetSRID(ST_MakeEnvelope(
+      -169.8046875, 21.289374355860424,
+      -47.4609375, 72.0739114882038
+  ), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailabletimespans_2010_2014_in_usa_extents;
+
+SELECT '2010 - 2014' NOT IN (SELECT timespan_id
+FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_SetSRID(ST_MakePoint(0, 0), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailabletimespans_no_usa_bg_not_in_zero_point;
+
+SELECT '2010 - 2014' IN (SELECT timespan_id
+FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, 'us.census.acs.B01003001'
+) WHERE valid_numer = True)
+AS _obs_getavailabletimespans_total_pop_in_2010_2014;
+
+SELECT '2010 - 2014' NOT IN (SELECT timespan_id
+FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, 'foo.bar.baz'
+) WHERE valid_numer = True)
+AS _obs_getavailabletimespans_foobarbaz_not_in_2010_2014;
+
+SELECT '2010 - 2014' IN (SELECT timespan_id
+FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, 'us.census.acs.B01003001'
+) WHERE valid_denom = True)
+AS _obs_getavailablegeometries_total_pop_denom_in_2010_2014;
+
+SELECT '2010 - 2014' NOT IN (SELECT timespan_id
+FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, 'foo.bar.baz'
+) WHERE valid_denom = True)
+AS _obs_getavailablegeometries_foobarbaz_denom_not_in_2010_2014;
+
+--
+-- _OBS_GetGeometryScores tests
+--
+
+SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
+       ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+             'us.census.tiger.county', 'us.census.tiger.zcta5']
+       AS _obs_geometryscores_500m_buffer
+       FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 500)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.county', 'us.census.tiger.zcta5'])
+      WHERE table_id LIKE '%2015%';
+
+SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
+       ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+             'us.census.tiger.zcta5', 'us.census.tiger.county']
+       AS _obs_geometryscores_5km_buffer
+       FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 5000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.county', 'us.census.tiger.zcta5'])
+      WHERE table_id LIKE '%2015%';
+
+SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
+       ARRAY['us.census.tiger.census_tract', 'us.census.tiger.block_group',
+             'us.census.tiger.zcta5', 'us.census.tiger.county']
+       AS _obs_geometryscores_50km_buffer
+       FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%';
+
+SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
+       ARRAY[ 'us.census.tiger.zcta5', 'us.census.tiger.census_tract',
+              'us.census.tiger.county', 'us.census.tiger.block_group' ]
+      AS _obs_geometryscores_500km_buffer
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 500000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%';
+
+SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
+       ARRAY['us.census.tiger.county', 'us.census.tiger.zcta5',
+             'us.census.tiger.census_tract', 'us.census.tiger.block_group']
+      AS _obs_geometryscores_2500km_buffer
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 2500000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%';
+
+SELECT JSON_Object_Agg(column_id, numgeoms::int ORDER BY numgeoms DESC)::Text
+      = '{ "us.census.tiger.block_group" : 9, "us.census.tiger.census_tract" : 3, "us.census.tiger.zcta5" : 0, "us.census.tiger.county" : 0 }'
+      AS _obs_geometryscores_numgeoms_500m_buffer
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 500)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%';
+
+SELECT JSON_Object_Agg(column_id, numgeoms::int ORDER BY numgeoms DESC)::Text =
+      '{ "us.census.tiger.block_group" : 880, "us.census.tiger.census_tract" : 310, "us.census.tiger.zcta5" : 45, "us.census.tiger.county" : 1 }'
+      AS _obs_geometryscores_numgeoms_5km_buffer
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 5000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%';
+
+SELECT JSON_Object_Agg(column_id, numgeoms::int ORDER BY numgeoms DESC)::Text =
+      '{ "us.census.tiger.block_group" : 11531, "us.census.tiger.census_tract" : 3601, "us.census.tiger.zcta5" : 550, "us.census.tiger.county" : 14 }'
+      AS _obs_geometryscores_numgeoms_50km_buffer
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%';
+
+SELECT JSON_Object_Agg(column_id, numgeoms::int ORDER BY numgeoms DESC)::Text =
+      '{ "us.census.tiger.block_group" : 48917, "us.census.tiger.census_tract" : 15969, "us.census.tiger.zcta5" : 6534, "us.census.tiger.county" : 314 }'
+      AS _obs_geometryscores_numgeoms_500km_buffer
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 500000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%';
+
+SELECT JSON_Object_Agg(column_id, numgeoms::int ORDER BY numgeoms DESC)::Text =
+      '{ "us.census.tiger.block_group" : 169191, "us.census.tiger.census_tract" : 56469, "us.census.tiger.zcta5" : 26525, "us.census.tiger.county" : 2753 }'
+      AS _obs_geometryscores_numgeoms_2500km_buffer
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 2500000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%';
+
+SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
+       ARRAY['us.census.tiger.county', 'us.census.tiger.zcta5',
+             'us.census.tiger.census_tract', 'us.census.tiger.block_group']
+      AS _obs_geometryscores_500km_buffer_50_geoms
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'], 50)
+      WHERE table_id LIKE '%2015%';
+
+SELECT ARRAY_AGG(column_id ORDER BY score DESC)
+      = ARRAY['us.census.tiger.zcta5', 'us.census.tiger.census_tract',
+              'us.census.tiger.county', 'us.census.tiger.block_group']
+      AS _obs_geometryscores_500km_buffer_500_geoms
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'], 500)
+      WHERE table_id LIKE '%2015%';
+
+SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
+       ARRAY['us.census.tiger.census_tract', 'us.census.tiger.block_group',
+             'us.census.tiger.zcta5', 'us.census.tiger.county']
+      AS _obs_geometryscores_500km_buffer_2500_geoms
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'], 2500)
+      WHERE table_id LIKE '%2015%';
+
+SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
+       ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+             'us.census.tiger.zcta5', 'us.census.tiger.county']
+      AS _obs_geometryscores_500km_buffer_25000_geoms
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'], 25000)
+      WHERE table_id LIKE '%2015%';
+
+--
+-- OBS_LegacyBuilderMetadata tests
+--
+
+SELECT 'us.census.acs.B01003001' IN (SELECT
+  (jsonb_array_elements(((jsonb_array_elements(subsection))->'f1')->'columns')->'f1')->>'id' AS id
+  FROM cdb_observatory.OBS_LegacyBuilderMetadata()
+) AS _total_pop_in_legacy_builder_metadata;
+
+SELECT 'us.census.acs.B19013001' IN (SELECT
+  (jsonb_array_elements(((jsonb_array_elements(subsection))->'f1')->'columns')->'f1')->>'id' AS id
+  FROM cdb_observatory.OBS_LegacyBuilderMetadata()
+) AS _median_income_in_legacy_builder_metadata;
+
+SELECT 'us.census.acs.B19083001' IN (SELECT
+  (jsonb_array_elements(((jsonb_array_elements(subsection))->'f1')->'columns')->'f1')->>'id' AS id
+  FROM cdb_observatory.OBS_LegacyBuilderMetadata()
+) AS _gini_in_legacy_builder_metadata;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT
+  (jsonb_array_elements(((jsonb_array_elements(subsection))->'f1')->'columns')->'f1')->>'id' AS id
+  FROM cdb_observatory.OBS_LegacyBuilderMetadata('sum')
+) AS _total_pop_in_legacy_builder_metadata_sums;
+
+SELECT 'us.census.acs.B19013001' IN (SELECT
+  (jsonb_array_elements(((jsonb_array_elements(subsection))->'f1')->'columns')->'f1')->>'id' AS id
+  FROM cdb_observatory.OBS_LegacyBuilderMetadata('sum')
+) AS _median_income_in_legacy_builder_metadata_sums;
+
+SELECT 'us.census.acs.B19083001' NOT IN (SELECT
+  (jsonb_array_elements(((jsonb_array_elements(subsection))->'f1')->'columns')->'f1')->>'id' AS id
+  FROM cdb_observatory.OBS_LegacyBuilderMetadata('sum')
+) AS _gini_not_in_legacy_builder_metadata_sums;
+
+SELECT COUNT(*) = 0 _no_dupe_subsections_in_legacy_builder_metadata FROM (
+  SELECT name, subsection, count(*) FROM
+    (SELECT name, ((JSONB_Array_Elements(subsection))->'f1')->>'id' subsection
+    FROM cdb_observatory.obs_legacybuildermetadata()) foo
+  GROUP BY name, subsection
+  HAVING count(*) > 1
+) bar;
+

src/pg/test/sql/44_observatory_geometries_test.sql

@@ -42,7 +42,7 @@ SELECT cdb_observatory.OBS_GetBoundary(
 SELECT cdb_observatory.OBS_GetBoundary(
   cdb_observatory._TestPoint(),
   'us.census.tiger.census_tract',
-  '2014'
+  '2015'
 ) = :'cartodb_census_tract_geometry' As OBS_GetBoundary_year_census_tract;
 
 -- should return null
@@ -65,7 +65,7 @@ SELECT cdb_observatory.OBS_GetBoundaryId(
 SELECT cdb_observatory.OBS_GetBoundaryId(
   cdb_observatory._TestPoint(),
   'us.census.tiger.census_tract',
-  '2014'
+  '2015'
 ) = '36047048500'::text As OBS_GetBoundaryId_cartodb_census_tract_with_year;
 
 -- should give back '36047', the geoid of cartodb's county (King's/
@@ -73,7 +73,7 @@ SELECT cdb_observatory.OBS_GetBoundaryId(
 SELECT cdb_observatory.OBS_GetBoundaryId(
   cdb_observatory._TestPoint(),
   'us.census.tiger.county',
-  '2014'
+  '2015'
 ) = '36047'::text As OBS_GetBoundaryId_cartodb_county_with_year;
 
 -- should give back null since there is not a census tract at null island
@@ -104,7 +104,7 @@ SELECT cdb_observatory.OBS_GetBoundaryById(
 SELECT cdb_observatory.OBS_GetBoundaryById(
   '36047',
   'us.census.tiger.county',
-  '2014'
+  '2015'
 ) = :'cartodb_county_geometry' OBS_GetBoundaryById_boundary_id_mismatch_geom_id;
 
 -- should give null since boundary_id does not match geometry reference id
@@ -115,6 +115,34 @@ SELECT cdb_observatory.OBS_GetBoundaryById(
 
 -- _OBS_GetBoundariesByGeometry
 
+SELECT array_agg(geom_refs) = Array[ '1104486618765', '1104486642837',
+                                '1104991798384', '1105044325367',
+                                '1105089330200', '1105089331758']
+As _OBS_GetBoundariesByGeometry_roads_around_cartodb
+FROM (
+  SELECT *
+  FROM cdb_observatory._OBS_GetBoundariesByGeometry(
+    -- near CartoDB's office
+    ST_MakeEnvelope(-74,40.69,-73.99,40.7,
+                    4326),
+    'us.census.tiger.prisecroads_geom')
+  ORDER BY geom_refs ASC
+) As m(the_geom, geom_refs);
+
+SELECT
+  array_agg(geom_refs) = Array['1102654301684', '1102654307106',
+                               '1102654326686', '1102654351507' ]
+As _OBS_GetBoundariesByGeometry_points_around_cartodb
+FROM (
+  SELECT *
+  FROM cdb_observatory._OBS_GetBoundariesByGeometry(
+    -- near CartoDB's office
+    ST_MakeEnvelope(-73.9452409744,40.6988851644,-73.9280319214,40.7101254524,
+                    4326),
+    'us.census.tiger.pointlm_geom')
+  ORDER BY geom_refs ASC
+) As m(the_geom, geom_refs);
+
 -- check that all census tracts intersecting with the geometry are returned
 -- order them to ensure that the same values are returned
 SELECT
@@ -270,7 +298,7 @@ FROM (
                     -73.9280319214,40.7101254524,
                     4326),
     'us.census.tiger.census_tract',
-    '2014')
+    '2015')
   ORDER BY geom_refs ASC
 ) As m(the_geom, geom_refs);
 
@@ -313,7 +341,7 @@ FROM (
     cdb_observatory._testpoint(),
     500,
     'us.census.tiger.census_tract',
-    '2014')
+    '2015')
   ORDER BY geom_refs ASC
 ) As m(the_geom, geom_refs);
 
@@ -330,21 +358,4 @@ FROM (
   ORDER BY geom_refs ASC
 ) As m(the_geom, geom_refs);
 
--- _OBS_GetGeometryMetadata
--- get metadata for census tracts
-
-SELECT
-  geoid_colname = 'geoid' As geoid_name_matches,
-  target_table = 'obs_fc050f0b8673cfe3c6aa1040f749eb40975691b7' As table_name_matches,
-  geom_colname = 'the_geom' As geom_name_matches
-FROM cdb_observatory._OBS_GetGeometryMetadata('us.census.tiger.census_tract')
-     As m(geoid_colname, target_table, geom_colname);
-
--- get metadata for boundaries with clipped geometries
- SELECT
-   geoid_colname = 'geoid' As geoid_name_matches,
-   target_table = 'obs_fcd4e4f5610f6764973ef8c0c215b2e80bec8963' As table_name_matches,
-   geom_colname = 'the_geom' As geom_name_matches
- FROM cdb_observatory._OBS_GetGeometryMetadata('us.census.tiger.census_tract_clipped') As m(geoid_colname, target_table, geom_colname);
-
 \i test/fixtures/drop_fixtures.sql

src/python/requirements.txt

@@ -0,0 +1,3 @@
+nose
+nose_parameterized
+psycopg2

src/python/test/README.md

@@ -0,0 +1,14 @@
+### Integration/performance tests
+
+Tests here are meant to be run on a box with an Observatory meta/data dump
+loaded and ready to be tested against the API.
+
+The local Python needs the requirements in `src/python/requirements.txt`.
+
+In order to find and access the correct database, the `PGUSER`, `PGPASSWORD`,
+`PGHOST`, `PGPORT` and `PGDATABASE` env variables should be set.
+
+Tests should be executed as follows:
+
+     nosetests test/autotest.py
+     nosetests -s test/perftest.py

src/python/test/autotest.py

@@ -0,0 +1,309 @@
+from nose.tools import assert_equal, assert_is_not_none
+from nose.plugins.skip import SkipTest
+from nose_parameterized import parameterized
+
+from util import query
+
+USE_SCHEMA = True
+
+MEASURE_COLUMNS = query('''
+SELECT distinct numer_id, Coalesce(numer_aggregate, '') NOT ILIKE 'sum' as point_only
+FROM observatory.obs_meta
+WHERE numer_type ILIKE 'numeric'
+AND numer_weight > 0
+''').fetchall()
+
+CATEGORY_COLUMNS = query('''
+SELECT distinct numer_id
+FROM observatory.obs_meta
+WHERE numer_type ILIKE 'text'
+AND numer_weight > 0
+''').fetchall()
+
+BOUNDARY_COLUMNS = query('''
+SELECT id FROM observatory.obs_column
+WHERE type ILIKE 'geometry'
+AND weight > 0
+''').fetchall()
+
+US_CENSUS_MEASURE_COLUMNS = query('''
+SELECT distinct numer_name
+FROM observatory.obs_meta
+WHERE numer_type ILIKE 'numeric'
+AND 'us.census.acs.acs' = ANY (subsection_tags)
+AND numer_weight > 0
+''').fetchall()
+
+
+SKIP_COLUMNS = set([
+    u'mx.inegi_columns.INDI18',
+    u'mx.inegi_columns.ECO40',
+    u'mx.inegi_columns.POB34',
+    u'mx.inegi_columns.POB63',
+    u'mx.inegi_columns.INDI7',
+    u'mx.inegi_columns.EDU28',
+    u'mx.inegi_columns.SCONY10',
+    u'mx.inegi_columns.EDU31',
+    u'mx.inegi_columns.POB7',
+    u'mx.inegi_columns.VIV30',
+    u'mx.inegi_columns.INDI12',
+    u'mx.inegi_columns.EDU13',
+    u'mx.inegi_columns.ECO43',
+    u'mx.inegi_columns.VIV9',
+    u'mx.inegi_columns.HOGAR25',
+    u'mx.inegi_columns.POB32',
+    u'mx.inegi_columns.ECO7',
+    u'mx.inegi_columns.INDI19',
+    u'mx.inegi_columns.INDI16',
+    u'mx.inegi_columns.POB65',
+    u'mx.inegi_columns.INDI3',
+    u'mx.inegi_columns.INDI9',
+    u'mx.inegi_columns.POB36',
+    u'mx.inegi_columns.POB33',
+    u'mx.inegi_columns.POB58',
+    u'mx.inegi_columns.DISC4',
+    u'mx.inegi_columns.VIV41',
+    u'mx.inegi_columns.VIV40',
+    u'mx.inegi_columns.VIV17',
+    u'mx.inegi_columns.VIV25',
+    u'mx.inegi_columns.EDU10',
+    u'whosonfirst.wof_disputed_name',
+    u'us.census.tiger.fullname',
+    u'whosonfirst.wof_marinearea_name',
+    u'us.census.tiger.mtfcc',
+    u'whosonfirst.wof_county_name',
+    u'whosonfirst.wof_region_name',
+])
+
+#def default_geometry_id(column_id):
+#    '''
+#    Returns default test point for the column_id.
+#    '''
+#    if column_id == 'whosonfirst.wof_disputed_geom':
+#        return 'ST_SetSRID(ST_MakePoint(76.57, 33.78), 4326)'
+#    elif column_id == 'whosonfirst.wof_marinearea_geom':
+#        return 'ST_SetSRID(ST_MakePoint(-68.47, 43.33), 4326)'
+#    elif column_id in ('us.census.tiger.school_district_elementary',
+#                       'us.census.tiger.school_district_secondary',
+#                       'us.census.tiger.school_district_elementary_clipped',
+#                       'us.census.tiger.school_district_secondary_clipped'):
+#        return 'ST_SetSRID(ST_MakePoint(-73.7067, 40.7025), 4326)'
+#    elif column_id.startswith('es.ine'):
+#        return 'ST_SetSRID(ST_MakePoint(-2.51141249535454, 42.8226119029222), 4326)'
+#    elif column_id.startswith('us.zillow'):
+#        return 'ST_SetSRID(ST_MakePoint(-81.3544048197256, 28.3305906291771), 4326)'
+#    elif column_id.startswith('ca.'):
+#        return ''
+#    else:
+#        return 'ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)'
+
+
+def default_lonlat(column_id):
+    '''
+    Returns default test point for the column_id.
+    '''
+    if column_id == 'whosonfirst.wof_disputed_geom':
+        return (76.57, 33.78)
+    elif column_id == 'whosonfirst.wof_marinearea_geom':
+        return (-68.47, 43.33)
+    elif column_id in ('us.census.tiger.school_district_elementary',
+                       'us.census.tiger.school_district_secondary',
+                       'us.census.tiger.school_district_elementary_clipped',
+                       'us.census.tiger.school_district_secondary_clipped'):
+        return (40.7025, -73.7067)
+    elif column_id.startswith('uk'):
+        if 'WA' in column_id:
+            return (51.46844551219723, -3.184833526611328)
+        else:
+            return (51.51461834694225, -0.08883476257324219)
+    elif column_id.startswith('es'):
+        return (42.8226119029222, -2.51141249535454)
+    elif column_id.startswith('us.zillow'):
+        return (28.3305906291771, -81.3544048197256)
+    elif column_id.startswith('mx.'):
+        return (19.41347699386547, -99.17019367218018)
+    elif column_id.startswith('th.'):
+        return (13.725377712079784, 100.49263000488281)
+    # cols for French Guyana only
+    elif column_id in ('fr.insee.P12_RP_CHOS', 'fr.insee.P12_RP_HABFOR'
+                       , 'fr.insee.P12_RP_EAUCH', 'fr.insee.P12_RP_BDWC'
+                       , 'fr.insee.P12_RP_MIDUR', 'fr.insee.P12_RP_CLIM'
+                       , 'fr.insee.P12_RP_MIBOIS', 'fr.insee.P12_RP_CASE'
+                       , 'fr.insee.P12_RP_TTEGOU', 'fr.insee.P12_RP_ELEC'
+                       , 'fr.insee.P12_ACTOCC15P_ILT45D'
+                       , 'fr.insee.P12_RP_CHOS', 'fr.insee.P12_RP_HABFOR'
+                       , 'fr.insee.P12_RP_EAUCH', 'fr.insee.P12_RP_BDWC'
+                       , 'fr.insee.P12_RP_MIDUR', 'fr.insee.P12_RP_CLIM'
+                       , 'fr.insee.P12_RP_MIBOIS', 'fr.insee.P12_RP_CASE'
+                       , 'fr.insee.P12_RP_TTEGOU', 'fr.insee.P12_RP_ELEC'
+                       , 'fr.insee.P12_ACTOCC15P_ILT45D'):
+        return (4.938408371206558, -52.32908248901367)
+    elif column_id.startswith('fr.'):
+        return (48.860875144709475, 2.3613739013671875)
+    elif column_id.startswith('ca.'):
+        return (43.65594991256823, -79.37965393066406)
+    elif column_id.startswith('us.census.'):
+        return (40.7, -73.9)
+    elif column_id.startswith('us.dma.'):
+        return (40.7, -73.9)
+    elif column_id.startswith('us.ihme.'):
+        return (40.7, -73.9)
+    elif column_id.startswith('us.bls.'):
+        return (40.7, -73.9)
+    elif column_id.startswith('us.qcew.'):
+        return (40.7, -73.9)
+    elif column_id.startswith('whosonfirst.'):
+        return (40.7, -73.9)
+    elif column_id.startswith('us.epa.'):
+        return (40.7, -73.9)
+    elif column_id.startswith('eu.'):
+        raise SkipTest('No tests for Eurostat!')
+    elif column_id.startswith('br.'):
+        return (-23.53, -46.63)
+    elif column_id.startswith('au.'):
+        return (-33.8806, 151.2131)
+    else:
+        raise Exception('No catalog point set for {}'.format(column_id))
+
+
+def default_point(column_id):
+    lat, lng = default_lonlat(column_id)
+    return 'ST_SetSRID(ST_MakePoint({lng}, {lat}), 4326)'.format(
+        lat=lat, lng=lng)
+
+
+def default_area(column_id):
+    '''
+    Returns default test area for the column_id
+    '''
+    point = default_point(column_id)
+    area = 'ST_Transform(ST_Buffer(ST_Transform({point}, 3857), 250), 4326)'.format(
+        point=point)
+    return area
+
+@parameterized(US_CENSUS_MEASURE_COLUMNS)
+def test_get_us_census_measure_points(name):
+    resp = query('''
+SELECT * FROM {schema}OBS_GetUSCensusMeasure({point}, '{name}')
+                 '''.format(name=name.replace("'", "''"),
+                            schema='cdb_observatory.' if USE_SCHEMA else '',
+                            point=default_point('')))
+    rows = resp.fetchall()
+    assert_equal(1, len(rows))
+    assert_is_not_none(rows[0][0])
+
+
+@parameterized(MEASURE_COLUMNS)
+def test_get_measure_areas(column_id, point_only):
+    if column_id in SKIP_COLUMNS:
+        raise SkipTest('Column {} should be skipped'.format(column_id))
+    if point_only:
+        return
+    resp = query('''
+SELECT * FROM {schema}OBS_GetMeasure({area}, '{column_id}')
+                 '''.format(column_id=column_id,
+                            schema='cdb_observatory.' if USE_SCHEMA else '',
+                            area=default_area(column_id)))
+    rows = resp.fetchall()
+    assert_equal(1, len(rows))
+    assert_is_not_none(rows[0][0])
+
+
+@parameterized(MEASURE_COLUMNS)
+def test_get_measure_points(column_id, point_only):
+    if column_id in SKIP_COLUMNS:
+        raise SkipTest('Column {} should be skipped'.format(column_id))
+    resp = query('''
+SELECT * FROM {schema}OBS_GetMeasure({point}, '{column_id}')
+                 '''.format(column_id=column_id,
+                            schema='cdb_observatory.' if USE_SCHEMA else '',
+                            point=default_point(column_id)))
+    rows = resp.fetchall()
+    assert_equal(1, len(rows))
+    assert_is_not_none(rows[0][0])
+
+#@parameterized(CATEGORY_COLUMNS)
+#def test_get_category_areas(column_id):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetCategory({area}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            area=default_area(column_id)))
+#    assert_equal(resp.status_code, 200)
+#    rows = resp.json()['rows']
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0][0])
+
+@parameterized(CATEGORY_COLUMNS)
+def test_get_category_points(column_id):
+    if column_id in SKIP_COLUMNS:
+        raise SkipTest('Column {} should be skipped'.format(column_id))
+    resp = query('''
+SELECT * FROM {schema}OBS_GetCategory({point}, '{column_id}')
+                 '''.format(column_id=column_id,
+                            schema='cdb_observatory.' if USE_SCHEMA else '',
+                            point=default_point(column_id)))
+    rows = resp.fetchall()
+    assert_equal(1, len(rows))
+    assert_is_not_none(rows[0][0])
+
+#@parameterized(BOUNDARY_COLUMNS)
+#def test_get_boundaries_by_geometry(column_id):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetBoundariesByGeometry({area}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            area=default_area(column_id)))
+#    assert_equal(resp.status_code, 200)
+#    rows = resp.json()['rows']
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0][0])
+
+#@parameterized(BOUNDARY_COLUMNS)
+#def test_get_points_by_geometry(column_id):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetPointsByGeometry({area}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            area=default_area(column_id)))
+#    assert_equal(resp.status_code, 200)
+#    rows = resp.json()['rows']
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0][0])
+
+#@parameterized(BOUNDARY_COLUMNS)
+#def test_get_boundary_points(column_id):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetBoundary({point}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            point=default_point(column_id)))
+#    assert_equal(resp.status_code, 200)
+#    rows = resp.json()['rows']
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0][0])
+
+#@parameterized(BOUNDARY_COLUMNS)
+#def test_get_boundary_id(column_id):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetBoundaryId({point}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            point=default_point(column_id)))
+#    assert_equal(resp.status_code, 200)
+#    rows = resp.json()['rows']
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0][0])
+
+#@parameterized(BOUNDARY_COLUMNS)
+#def test_get_boundary_by_id(column_id):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetBoundaryById({geometry_id}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            geometry_id=default_geometry_id(column_id)))
+#    assert_equal(resp.status_code, 200)
+#    rows = resp.json()['rows']
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0][0])
+

src/python/test/perftest.py

@@ -0,0 +1,350 @@
+from nose.tools import assert_equal, assert_is_not_none
+from nose_parameterized import parameterized
+
+from util import query, commit
+
+from time import time
+
+import json
+import os
+
+USE_SCHEMA = True
+
+for q in (
+        'DROP TABLE IF EXISTS obs_perftest_simple',
+        '''CREATE TABLE obs_perftest_simple (cartodb_id SERIAL PRIMARY KEY,
+           point GEOMETRY,
+           geom GEOMETRY,
+           offset_geom GEOMETRY,
+           name TEXT, measure NUMERIC, category TEXT)''',
+        '''INSERT INTO obs_perftest_simple (point, geom, offset_geom, name)
+           SELECT ST_PointOnSurface(the_geom) point,
+                  the_geom geom,
+                  ST_Translate(the_geom, -0.1, 0.1) offset_geom,
+                  geom_refs AS name
+           FROM (SELECT * FROM {schema}OBS_GetBoundariesByGeometry(
+                 st_makeenvelope(-74.1, 40.5,
+                                 -73.8, 40.9, 4326),
+                 'us.census.tiger.census_tract_clipped')) foo
+           ORDER BY ST_NPoints(the_geom) ASC
+           LIMIT 1000''',
+        'DROP TABLE IF EXISTS obs_perftest_complex',
+        '''CREATE TABLE obs_perftest_complex (cartodb_id SERIAL PRIMARY KEY,
+           point GEOMETRY,
+           geom GEOMETRY,
+           offset_geom GEOMETRY,
+           name TEXT, measure NUMERIC, category TEXT)''',
+        '''INSERT INTO obs_perftest_complex (point, geom, offset_geom, name)
+           SELECT ST_PointOnSurface(the_geom) point,
+                  the_geom geom,
+                  ST_Translate(the_geom, -0.1, 0.1) offset_geom,
+                  geom_refs AS name
+           FROM (SELECT * FROM {schema}OBS_GetBoundariesByGeometry(
+                 st_makeenvelope(-75.05437469482422,40.66319159533881,
+                                 -73.81885528564453,41.745696344339564, 4326),
+                 'us.census.tiger.county_clipped')) foo
+           ORDER BY ST_NPoints(the_geom) DESC
+           LIMIT 50;''',
+        'DROP TABLE IF EXISTS obs_perftest_country_simple',
+        '''CREATE TABLE obs_perftest_country_simple (cartodb_id SERIAL PRIMARY KEY,
+           geom GEOMETRY,
+           name TEXT) ''',
+        '''INSERT INTO obs_perftest_country_simple (geom, name)
+           SELECT the_geom geom,
+                  geom_refs AS name
+           FROM (SELECT * FROM {schema}OBS_GetBoundariesByGeometry(
+                 st_makeenvelope(-179,-89, 179,89, 4326),
+                 'whosonfirst.wof_country_geom')) foo
+           ORDER BY ST_NPoints(the_geom) ASC
+           LIMIT 50;''',
+        'DROP TABLE IF EXISTS obs_perftest_country_complex',
+        '''CREATE TABLE obs_perftest_country_complex (cartodb_id SERIAL PRIMARY KEY,
+           geom GEOMETRY,
+           name TEXT) ''',
+        '''INSERT INTO obs_perftest_country_complex (geom, name)
+           SELECT the_geom geom,
+                  geom_refs AS name
+           FROM (SELECT * FROM {schema}OBS_GetBoundariesByGeometry(
+                 st_makeenvelope(-179,-89, 179,89, 4326),
+                 'whosonfirst.wof_country_geom')) foo
+           ORDER BY ST_NPoints(the_geom) DESC
+           LIMIT 50;''',
+        #'''SET statement_timeout = 5000;'''
+):
+    q_formatted = q.format(
+        schema='cdb_observatory.' if USE_SCHEMA else '',
+    )
+    resp = query(q_formatted)
+    if q.lower().startswith('insert'):
+        if resp.rowcount == 0:
+            raise Exception('''Performance fixture creation "{}" inserted 0 rows,
+                            this will break tests.  Check the query to determine
+                            what is going wrong.'''.format(q_formatted))
+    commit()
+
+
+ARGS = {
+    ('OBS_GetMeasureByID', None): "name, 'us.census.acs.B01001002', '{}'",
+    ('OBS_GetMeasure', 'predenominated'): "{}, 'us.census.acs.B01003001', null, {}",
+    ('OBS_GetMeasure', 'area'): "{}, 'us.census.acs.B01001002', 'area', {}",
+    ('OBS_GetMeasure', 'denominator'): "{}, 'us.census.acs.B01001002', 'denominator', {}",
+    ('OBS_GetCategory', None): "{}, 'us.census.spielman_singleton_segments.X10', {}",
+    ('_OBS_GetGeometryScores', None): "{}, NULL"
+}
+
+
+def record(params, results):
+    sha = os.environ['OBS_EXTENSION_SHA']
+    msg = os.environ.get('OBS_EXTENSION_MSG')
+    fpath = os.path.join(os.environ['OBS_PERFTEST_DIR'], sha + '.json')
+    if os.path.isfile(fpath):
+        tests = json.load(open(fpath, 'r'))
+    else:
+        tests = {}
+    with open(fpath, 'w') as fhandle:
+        tests[json.dumps(params)] = {
+            'params': params,
+            'results': results
+        }
+        json.dump(tests, fhandle)
+
+@parameterized([
+    ('simple', '_OBS_GetGeometryScores', 'NULL', 1),
+    ('simple', '_OBS_GetGeometryScores', 'NULL', 500),
+    ('simple', '_OBS_GetGeometryScores', 'NULL', 3000),
+
+    ('complex', '_OBS_GetGeometryScores', 'NULL', 1),
+    ('complex', '_OBS_GetGeometryScores', 'NULL', 500),
+    ('complex', '_OBS_GetGeometryScores', 'NULL', 3000),
+
+    ('country_simple', '_OBS_GetGeometryScores', 'NULL', 1),
+    ('country_simple', '_OBS_GetGeometryScores', 'NULL', 500),
+    ('country_simple', '_OBS_GetGeometryScores', 'NULL', 5000),
+
+    ('country_complex', '_OBS_GetGeometryScores', 'NULL', 1),
+    ('country_complex', '_OBS_GetGeometryScores', 'NULL', 500),
+    ('country_complex', '_OBS_GetGeometryScores', 'NULL', 5000),
+])
+def test_getgeometryscores_performance(geom_complexity, api_method, filters, target_geoms):
+    print api_method, geom_complexity, filters, target_geoms
+
+    rownums = (1, 5, 10, ) if 'complex' in geom_complexity else (5, 25, 50,)
+    results = []
+    for rows in rownums:
+        stmt = '''SELECT {schema}{api_method}(geom, {filters}, {target_geoms})
+                   FROM obs_perftest_{complexity}
+                   WHERE cartodb_id <= {n}'''.format(
+                       complexity=geom_complexity,
+                       schema='cdb_observatory.' if USE_SCHEMA else '',
+                       api_method=api_method,
+                       filters=filters,
+                       target_geoms=target_geoms,
+                       n=rows)
+        start = time()
+        query(stmt)
+        end = time()
+        qps = (rows / (end - start))
+        results.append({
+            'rows': rows,
+            'qps': qps,
+            'stmt': stmt
+        })
+        print rows, ': ', qps, ' QPS'
+
+    if 'OBS_RECORD_TEST' in os.environ:
+        record({
+            'geom_complexity': geom_complexity,
+            'api_method': api_method,
+            'filters': filters,
+            'target_geoms': target_geoms
+        }, results)
+
+@parameterized([
+    ('simple', 'OBS_GetMeasureByID', None, 'us.census.tiger.census_tract', None),
+    ('complex', 'OBS_GetMeasureByID', None, 'us.census.tiger.county', None),
+
+    ('simple', 'OBS_GetMeasure', 'predenominated', 'point', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'predenominated', 'geom', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'predenominated', 'offset_geom', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'area', 'point', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'area', 'geom', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'area', 'offset_geom', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'denominator', 'point', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'denominator', 'geom', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'denominator', 'offset_geom', 'NULL'),
+    ('simple', 'OBS_GetCategory', None, 'point', 'NULL'),
+    ('simple', 'OBS_GetCategory', None, 'geom', 'NULL'),
+    ('simple', 'OBS_GetCategory', None, 'offset_geom', 'NULL'),
+
+    ('simple', 'OBS_GetMeasure', 'predenominated', 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'predenominated', 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'predenominated', 'offset_geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'area', 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'area', 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'area', 'offset_geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'denominator', 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'denominator', 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'denominator', 'offset_geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetCategory', None, 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetCategory', None, 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetCategory', None, 'offset_geom', "'us.census.tiger.census_tract'"),
+
+    ('complex', 'OBS_GetMeasure', 'predenominated', 'point', 'NULL'),
+    ('complex', 'OBS_GetMeasure', 'predenominated', 'geom', 'NULL'),
+    ('complex', 'OBS_GetMeasure', 'predenominated', 'offset_geom', 'NULL'),
+    ('complex', 'OBS_GetMeasure', 'area', 'point', 'NULL'),
+    ('complex', 'OBS_GetMeasure', 'area', 'geom', 'NULL'),
+    ('complex', 'OBS_GetMeasure', 'area', 'offset_geom', 'NULL'),
+    ('complex', 'OBS_GetMeasure', 'denominator', 'point', 'NULL'),
+    ('complex', 'OBS_GetMeasure', 'denominator', 'geom', 'NULL'),
+    ('complex', 'OBS_GetMeasure', 'denominator', 'offset_geom', 'NULL'),
+    ('complex', 'OBS_GetCategory', None, 'point', 'NULL'),
+    ('complex', 'OBS_GetCategory', None, 'geom', 'NULL'),
+    ('complex', 'OBS_GetCategory', None, 'offset_geom', 'NULL'),
+
+    ('complex', 'OBS_GetMeasure', 'predenominated', 'point', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetMeasure', 'predenominated', 'geom', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetMeasure', 'predenominated', 'offset_geom', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetMeasure', 'area', 'point', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetMeasure', 'area', 'geom', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetMeasure', 'area', 'offset_geom', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetMeasure', 'denominator', 'point', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetMeasure', 'denominator', 'geom', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetMeasure', 'denominator', 'offset_geom', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetCategory', None, 'point', "'us.census.tiger.census_tract'"),
+    ('complex', 'OBS_GetCategory', None, 'geom', "'us.census.tiger.census_tract'"),
+    ('complex', 'OBS_GetCategory', None, 'offset_geom', "'us.census.tiger.census_tract'"),
+])
+def test_getmeasure_performance(geom_complexity, api_method, normalization, geom, boundary):
+    print api_method, geom_complexity, normalization, geom, boundary
+    col = 'measure' if 'measure' in api_method.lower() else 'category'
+    results = []
+
+    rownums = (1, 5, 10, ) if geom_complexity == 'complex' else (5, 25, 50, )
+    for rows in rownums:
+        stmt = '''UPDATE obs_perftest_{complexity}
+                   SET {col} = {schema}{api_method}({args})
+                   WHERE cartodb_id <= {n}'''.format(
+                       col=col,
+                       complexity=geom_complexity,
+                       schema='cdb_observatory.' if USE_SCHEMA else '',
+                       api_method=api_method,
+                       args=ARGS[api_method, normalization].format(geom, boundary),
+                       n=rows)
+        start = time()
+        query(stmt)
+        end = time()
+        qps = (rows / (end - start))
+        results.append({
+            'rows': rows,
+            'qps': qps,
+            'stmt': stmt
+        })
+        print rows, ': ', qps, ' QPS'
+
+    if 'OBS_RECORD_TEST' in os.environ:
+        record({
+            'geom_complexity': geom_complexity,
+            'api_method': api_method,
+            'normalization': normalization,
+            'boundary': boundary,
+            'geom': geom
+        }, results)
+
+
+@parameterized([
+    ('simple', 'predenominated', 'point', 'null'),
+    ('simple', 'predenominated', 'geom', 'null'),
+    ('simple', 'predenominated', 'offset_geom', 'null'),
+    ('simple', 'area', 'point', 'null'),
+    ('simple', 'area', 'geom', 'null'),
+    ('simple', 'area', 'offset_geom', 'null'),
+    ('simple', 'denominator', 'point', 'null'),
+    ('simple', 'denominator', 'geom', 'null'),
+    ('simple', 'denominator', 'offset_geom', 'null'),
+
+    ('simple', 'predenominated', 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'predenominated', 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'predenominated', 'offset_geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'area', 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'area', 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'area', 'offset_geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'denominator', 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'denominator', 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'denominator', 'offset_geom', "'us.census.tiger.census_tract'"),
+
+    ('complex', 'predenominated', 'point', 'null'),
+    ('complex', 'predenominated', 'geom', 'null'),
+    ('complex', 'predenominated', 'offset_geom', 'null'),
+    ('complex', 'area', 'point', 'null'),
+    ('complex', 'area', 'geom', 'null'),
+    ('complex', 'area', 'offset_geom', 'null'),
+    ('complex', 'denominator', 'point', 'null'),
+    ('complex', 'denominator', 'geom', 'null'),
+    ('complex', 'denominator', 'offset_geom', 'null'),
+
+    ('complex', 'predenominated', 'point', "'us.census.tiger.county'"),
+    ('complex', 'predenominated', 'geom', "'us.census.tiger.county'"),
+    ('complex', 'predenominated', 'offset_geom', "'us.census.tiger.county'"),
+    ('complex', 'area', 'point', "'us.census.tiger.county'"),
+    ('complex', 'area', 'geom', "'us.census.tiger.county'"),
+    ('complex', 'area', 'offset_geom', "'us.census.tiger.county'"),
+    ('complex', 'denominator', 'point', "'us.census.tiger.county'"),
+    ('complex', 'denominator', 'geom', "'us.census.tiger.county'"),
+    ('complex', 'denominator', 'offset_geom', "'us.census.tiger.county'"),
+])
+def test_getmeasure_split_performance(geom_complexity, normalization, geom, boundary):
+    print geom_complexity, normalization, geom, boundary
+    results = []
+
+    rownums = (1, 5, 10, ) if geom_complexity == 'complex' else (10, 50, 100)
+    for rows in rownums:
+        stmt = '''
+with data as (
+  SELECT id, data FROM {schema}OBS_GetData(
+    (SELECT array_agg(({geom}, cartodb_id)::geomval)
+     FROM obs_perftest_{complexity}
+     WHERE cartodb_id <= {n}),
+    (SELECT {schema}OBS_GetMeta(
+      (SELECT st_setsrid(st_extent({geom}), 4326)
+       FROM obs_perftest_{complexity}
+       WHERE cartodb_id <= {n}),
+      '[{{
+          "numer_id": "us.census.acs.B01001002",
+          "normalization": "{normalization}",
+          "geom_id": {boundary}
+        }}]'::JSON
+    ))
+  ))
+UPDATE obs_perftest_{complexity}
+SET measure = (data->0->>'value')::Numeric
+FROM data
+WHERE obs_perftest_{complexity}.cartodb_id = data.id
+;
+        '''.format(
+            point_or_poly='point' if geom == 'point' else 'polygon',
+            complexity=geom_complexity,
+            schema='cdb_observatory.' if USE_SCHEMA else '',
+            normalization=normalization,
+            geom=geom,
+            boundary=boundary.replace("'", '"'),
+            n=rows)
+        start = time()
+        query(stmt)
+        end = time()
+        qps = (rows / (end - start))
+        results.append({
+            'rows': rows,
+            'qps': qps,
+            'stmt': stmt
+        })
+        print rows, ': ', qps, ' QPS'
+
+    if 'OBS_RECORD_TEST' in os.environ:
+        record({
+            'geom_complexity': geom_complexity,
+            'api_method': 'OBS_GetData',
+            'normalization': normalization,
+            'boundary': boundary,
+            'geom': geom
+        }, results)

src/python/test/util.py

@@ -0,0 +1,31 @@
+import os
+import psycopg2
+
+DB_CONN = psycopg2.connect('postgres://{user}:{password}@{host}:{port}/{database}'.format(
+    user=os.environ.get('PGUSER', 'postgres'),
+    password=os.environ.get('PGPASSWORD', ''),
+    host=os.environ.get('PGHOST', 'localhost'),
+    port=os.environ.get('PGPORT', '5432'),
+    database=os.environ.get('PGDATABASE', 'postgres'),
+))
+CURSOR = DB_CONN.cursor()
+
+
+def query(q):
+    '''
+    Query the database.
+    '''
+    try:
+        CURSOR.execute(q)
+        return CURSOR
+    except:
+        DB_CONN.rollback()
+        raise
+
+
+def commit():
+    try:
+        DB_CONN.commit()
+    except:
+        DB_CONN.rollback()
+        raise