Remove paragraph from RELEASE.md doc

Remove paragraph about generating upgrade and downgrade paths, as we're
not applying it to the release process.

NEWS.md

@@ -1,3 +1,54 @@
-0.0.1 (open date)
+1.0.0 (6/27/2016)
+-----
+
+* Incremented to 1.0.0 to be in compliance with [SemVer](http://semver.org/),
+  which disallows use of 0.x.x versions.  This also reflects that we are
+  already in production.
+
+__API Changes__
+
+* Added `OBS_DumpVersion` to look up version data ([#118](https://github.com/CartoDB/observatory-extension/pull/118))
+
+__Improvements__
+
+* Whether data exists for a geom now determined by polygon intersection instead of
+  BBOX overlap ([#119](https://github.com/CartoDB/observatory-extension/pull/119))
+* Automated tests cover Spanish and UK data
+  ([#115](https://github.com/CartoDB/observatory-extension/pull/115))
+* Automated tests cover `OBS_GetUSCensusMeasure`
+  ([#105](https://github.com/CartoDB/observatory-extension/pull/105))
+
+__Bugfixes__
+
+* Geom table can have different `geomref_colname` than the data table
+  ([#123](https://github.com/CartoDB/observatory-extension/pull/123))
+
+
+0.0.5 (5/27/2016)
+-----
+* Adds new function `OBS_GetMeasureById` ([#96](https://github.com/CartoDB/observatory-extension/pull/96))
+
+0.0.4 (5/25/2016)
+-----
+* Updates queries involving US Census measure tags to be more generic ([#95](https://github.com/CartoDB/observatory-extension/pull/95))
+* Fixes tests which relied on an erroneous subset of block groups ([#95](https://github.com/CartoDB/observatory-extension/pull/95))
+
+0.0.3 (5/24/2016)
+-----
+* Generalizes internal queries to properly pull from multiple named geometry references
+* Adds tests for Who's on First boundaries
+* Improves automatic fixtures testing script
+
+0.0.2 (5/19/2016)
+-----
+* Adds Data Observatory exploration functions
+* Adds Data Observatory boundary functions
+* Adds Data Observatory measure functions
+* Adds script to generate fixtures for tests
+* Adds script for the automatic testing of metadata
+* Adds full documentation for all included functions
+* removes `cartodb` extension dependency
+
+0.0.1 (5/19/2016)
 ------------------
-* First iteration of `OBS_GetDemographicSnapshot(location Geometry(Point,4326))`;
+* First iteration of `OBS_GetDemographicSnapshot(location Geometry(Point,4326))`

RELEASE.md

@@ -20,12 +20,6 @@ script for the new release, `release/observatory--X.Y.Z.sql`:
 make release
 ```
 
-Then, the release manager shall produce upgrade and downgrade scripts
-to migrate to/from the previous release. In the case of minor/patch
-releases this simply consist in extracting the functions that have changed
-and placing them in the proper `release/observatory--X.Y.Z--A.B.C.sql`
-file.
-
 The new release can be deployed for staging/smoke tests with this command:
 
 ```

doc/license.md

@@ -14,5 +14,5 @@ Natural Earth  |  [http://www.naturalearthdata.com/about/terms-of-use/](http://w
 Quattroshapes  |  [https://github.com/foursquare/quattroshapes/blob/master/LICENSE.md](https://github.com/foursquare/quattroshapes/blob/master/LICENSE.md)
 Zetashapes  |  [http://zetashapes.com/license](http://zetashapes.com/license)
 Spielman & Singleton  |  [https://www.openicpsr.org/repoEntity/show/41329](https://www.openicpsr.org/repoEntity/show/41329)
-Instituto Nacional de Estadistica  |  [http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout](http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout)
+El Instituto Nacional de Estadística (INE) | The National Statistics Institute (INE) of Spain includes data from multiple sources. If you are re-using their data, they explicitly require that you reference them accordingly<br /><br />[http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout](http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout)
 

doc/measures_functions.md

@@ -134,6 +134,40 @@ SET household_count = OBS_GetMeasure(the_geom, 'us.census.acs.B11001001')
 
 * If an unrecognized normalization type is input, raise an error: `'Only valid inputs for "normalize" are "area" (default) and "denominator".`
 
+## OBS_GetMeasureById(geom_ref text, measure_id text, boundary_id text)
+
+The ```OBS_GetMeasureById(geom_ref, measure_id, boundary_id)``` function returns any Data Observatory measure that corresponds to the boundary in ```boundary_id``` that has a geometry reference of ```geom_ref```.
+
+#### Arguments
+
+Name |Description
+--- | ---
+geom_ref | a geometry reference (e.g., a US Census geoid)
+measure_id | a measure identifier from the Data Observatory ([see available measures](https://cartodb.github.io/bigmetadata/observatory.pdf))  
+boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
+time_span (optional) | time span of interest (e.g., 2010 - 2014). If `NULL` is passed, the measure from the most recent data will be used.
+
+#### Returns
+
+A NUMERIC value
+
+Key | Description
+--- | ---
+value | the raw measure associated with `geom_ref`
+
+#### Example
+
+Add a measure to an empty column based on county geoids in your table
+
+```SQL
+UPDATE tablename
+SET household_count = OBS_GetMeasureById(geoid_column, 'us.census.acs.B11001001', 'us.census.tiger.county')
+```
+
+#### Errors
+
+* Returns `NULL` if there is a mismatch between the geometry reference and the boundary id such as using the geoid of a county with the boundary of block groups
+
 ## OBS_GetCategory(point geometry, category_id text)
 
 The ```OBS_GetCategory(point, category_id)``` function returns any Data Observatory Category value at a point location. The Categories available are currently limited to Segmentation categories. See the Segmentation section of the [Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf) for more detail.

release/observatory.control

@@ -1,5 +1,5 @@
 comment = 'CartoDB Observatory backend extension'
-default_version = '0.0.4'
+default_version = '1.0.0'
 requires = 'postgis'
 superuser = true
 schema = cdb_observatory

scripts/autotest.py

@@ -24,8 +24,9 @@ def query(q, is_meta=False, **options):
     params['api_key'] = META_API_KEY if is_meta else API_KEY
     return requests.get(url, params=params)
 
-MEASURE_COLUMNS = [(r['id'], ) for r in query('''
-SELECT id FROM obs_column
+MEASURE_COLUMNS = [(r['id'], r['point_only'], ) for r in query('''
+SELECT id, aggregate NOT ILIKE 'sum' as point_only
+FROM obs_column
 WHERE type ILIKE 'numeric'
 AND weight > 0
 ''', is_meta=True).json()['rows']]
@@ -42,7 +43,16 @@ WHERE type ILIKE 'geometry'
 AND weight > 0
 ''', is_meta=True).json()['rows']]
 
-def default_point(column_id):
+US_CENSUS_MEASURE_COLUMNS = [(r['name'], ) for r in query('''
+SELECT c.name FROM obs_column c, obs_column_tag ct
+WHERE type ILIKE 'numeric'
+AND c.id = ct.column_id
+AND ct.tag_id LIKE 'us.census%'
+AND weight > 0
+''', is_meta=True).json()['rows']]
+
+
+def default_geometry_id(column_id):
     '''
     Returns default test point for the column_id.
     '''
@@ -63,8 +73,71 @@ def default_point(column_id):
         return 'CDB_LatLng(40.7, -73.9)'
 
 
+def default_point(column_id):
+    '''
+    Returns default test point for the column_id.
+    '''
+    if column_id == 'whosonfirst.wof_disputed_geom':
+        return 'CDB_LatLng(33.78, 76.57)'
+    elif column_id == 'whosonfirst.wof_marinearea_geom':
+        return 'CDB_LatLng(43.33, -68.47)'
+    elif column_id in ('us.census.tiger.school_district_elementary',
+                       'us.census.tiger.school_district_secondary',
+                       'us.census.tiger.school_district_elementary_clipped',
+                       'us.census.tiger.school_district_secondary_clipped'):
+        return 'CDB_LatLng(40.7025, -73.7067)'
+    elif column_id.startswith('uk'):
+        if 'WA' in column_id:
+            return 'CDB_LatLng(51.46844551219723, -3.184833526611328)'
+        else:
+            return 'CDB_LatLng(51.51461834694225, -0.08883476257324219)'
+    elif column_id.startswith('es'):
+        return 'CDB_LatLng(42.8226119029222, -2.51141249535454)'
+    elif column_id.startswith('us.zillow'):
+        return 'CDB_LatLng(28.3305906291771, -81.3544048197256)'
+    else:
+        return 'CDB_LatLng(40.7, -73.9)'
+
+
+def default_area(column_id):
+    '''
+    Returns default test area for the column_id
+    '''
+    point = default_point(column_id)
+    area = 'ST_Transform(ST_Buffer(ST_Transform({point}, 3857), 1000), 4326)'.format(
+        point=point)
+    return area
+
+@parameterized(US_CENSUS_MEASURE_COLUMNS)
+def test_get_us_census_measure_points(name):
+    resp = query('''
+SELECT * FROM {schema}OBS_GetUSCensusMeasure({point}, '{name}')
+                 '''.format(name=name.replace("'", "''"),
+                            schema='cdb_observatory.' if USE_SCHEMA else '',
+                            point=default_point('')))
+    assert_equal(resp.status_code, 200)
+    rows = resp.json()['rows']
+    assert_equal(1, len(rows))
+    assert_is_not_none(rows[0].values()[0])
+
+
 @parameterized(MEASURE_COLUMNS)
-def test_measure_points(column_id):
+def test_get_measure_areas(column_id, point_only):
+    if point_only:
+        return
+    resp = query('''
+SELECT * FROM {schema}OBS_GetMeasure({area}, '{column_id}')
+                 '''.format(column_id=column_id,
+                            schema='cdb_observatory.' if USE_SCHEMA else '',
+                            area=default_area(column_id)))
+    assert_equal(resp.status_code, 200)
+    rows = resp.json()['rows']
+    assert_equal(1, len(rows))
+    assert_is_not_none(rows[0].values()[0])
+
+
+@parameterized(MEASURE_COLUMNS)
+def test_get_measure_points(column_id, point_only):
     resp = query('''
 SELECT * FROM {schema}OBS_GetMeasure({point}, '{column_id}')
                  '''.format(column_id=column_id,
@@ -75,8 +148,20 @@ SELECT * FROM {schema}OBS_GetMeasure({point}, '{column_id}')
     assert_equal(1, len(rows))
     assert_is_not_none(rows[0].values()[0])
 
+#@parameterized(CATEGORY_COLUMNS)
+#def test_get_category_areas(column_id):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetCategory({area}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            area=default_area(column_id)))
+#    assert_equal(resp.status_code, 200)
+#    rows = resp.json()['rows']
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0].values()[0])
+
 @parameterized(CATEGORY_COLUMNS)
-def test_category_points(column_id):
+def test_get_category_points(column_id):
     resp = query('''
 SELECT * FROM {schema}OBS_GetCategory({point}, '{column_id}')
                  '''.format(column_id=column_id,
@@ -87,14 +172,62 @@ SELECT * FROM {schema}OBS_GetCategory({point}, '{column_id}')
     assert_equal(1, len(rows))
     assert_is_not_none(rows[0].values()[0])
 
-@parameterized(BOUNDARY_COLUMNS)
-def test_boundary_points(column_id):
-    resp = query('''
-SELECT * FROM {schema}OBS_GetBoundary({point}, '{column_id}')
-                 '''.format(column_id=column_id,
-                            schema='cdb_observatory.' if USE_SCHEMA else '',
-                            point=default_point(column_id)))
-    assert_equal(resp.status_code, 200)
-    rows = resp.json()['rows']
-    assert_equal(1, len(rows))
-    assert_is_not_none(rows[0].values()[0])
+#@parameterized(BOUNDARY_COLUMNS)
+#def test_get_boundaries_by_geometry(column_id):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetBoundariesByGeometry({area}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            area=default_area(column_id)))
+#    assert_equal(resp.status_code, 200)
+#    rows = resp.json()['rows']
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0].values()[0])
+
+#@parameterized(BOUNDARY_COLUMNS)
+#def test_get_points_by_geometry(column_id):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetPointsByGeometry({area}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            area=default_area(column_id)))
+#    assert_equal(resp.status_code, 200)
+#    rows = resp.json()['rows']
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0].values()[0])
+
+#@parameterized(BOUNDARY_COLUMNS)
+#def test_get_boundary_points(column_id):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetBoundary({point}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            point=default_point(column_id)))
+#    assert_equal(resp.status_code, 200)
+#    rows = resp.json()['rows']
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0].values()[0])
+
+#@parameterized(BOUNDARY_COLUMNS)
+#def test_get_boundary_id(column_id):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetBoundaryId({point}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            point=default_point(column_id)))
+#    assert_equal(resp.status_code, 200)
+#    rows = resp.json()['rows']
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0].values()[0])
+
+#@parameterized(BOUNDARY_COLUMNS)
+#def test_get_boundary_by_id(column_id):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetBoundaryById({geometry_id}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            geometry_id=default_geometry_id(column_id)))
+#    assert_equal(resp.status_code, 200)
+#    rows = resp.json()['rows']
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0].values()[0])

scripts/generate_fixtures.py

@@ -33,7 +33,8 @@ def select_star(tablename):
 
 cdb = Dumpr('observatory.cartodb.com','')
 
-metadata = ['obs_table', 'obs_column_table', 'obs_column', 'obs_column_tag', 'obs_tag', 'obs_column_to_column']
+metadata = ['obs_table', 'obs_column_table', 'obs_column', 'obs_column_tag',
+            'obs_tag', 'obs_column_to_column', 'obs_dump_version']
 
 fixtures = [
     ('us.census.tiger.census_tract', 'us.census.tiger.census_tract', '2014'),

src/pg/observatory.control

@@ -1,5 +1,5 @@
 comment = 'CartoDB Observatory backend extension'
-default_version = '0.0.4'
+default_version = '1.0.0'
 requires = 'postgis'
 superuser = true
 schema = cdb_observatory

src/pg/sql/40_observatory_utility.sql

@@ -199,3 +199,20 @@ BEGIN
   RETURN result;
 END;
 $$ LANGUAGE plpgsql;
+
+-- Function that returns the currently deployed obs_dump_version from the
+-- remote table of the same name.
+
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_DumpVersion(
+)
+  RETURNS TEXT
+AS $$
+DECLARE
+  result text;
+BEGIN
+  EXECUTE '
+    SELECT MAX(dump_id) FROM observatory.obs_dump_version
+  ' INTO result;
+  RETURN result;
+END;
+$$ LANGUAGE plpgsql;

src/pg/sql/41_observatory_augmentation.sql

@@ -380,6 +380,53 @@ END;
 $$ LANGUAGE plpgsql;
 
 
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetMeasureById(
+  geom_ref TEXT,
+  measure_id TEXT,
+  boundary_id TEXT,
+  time_span TEXT DEFAULT NULL
+)
+RETURNS NUMERIC
+AS $$
+DECLARE
+  target_table TEXT;
+  colname TEXT;
+  measure_val NUMERIC;
+  data_geoid_colname TEXT;
+  test_query TEXT;
+BEGIN
+
+  SELECT x ->> 'colname', x ->> 'tablename' INTO colname, target_table
+  FROM cdb_observatory._OBS_GetColumnData(boundary_id, Array[measure_id], time_span) As x;
+
+  EXECUTE
+    format('SELECT ct.colname
+              FROM observatory.obs_column_to_column c2c,
+                   observatory.obs_column_table ct,
+                   observatory.obs_table t
+             WHERE c2c.reltype = ''geom_ref''
+               AND ct.column_id = c2c.source_id
+               AND ct.table_id = t.id
+               AND t.tablename = %L'
+     , target_table)
+  INTO data_geoid_colname;
+
+  RAISE DEBUG 'target_table %, colname %', target_table, colname;
+
+  EXECUTE format(
+      'SELECT %I
+       FROM observatory.%I
+       WHERE %I.%I = %L',
+       colname,
+       target_table,
+       target_table, data_geoid_colname, geom_ref)
+  INTO measure_val;
+
+  RETURN measure_val;
+
+END;
+$$ LANGUAGE plpgsql;
+
 CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetCategory(
   geom geometry(Geometry, 4326),
   category_id TEXT,
@@ -586,7 +633,7 @@ BEGIN
   q := q || q_select || format('FROM observatory.%I ', ((data_table_info)[1]->>'tablename'));
 
   q := format(q || ' ) ' || q_sum || ' ]::numeric[] FROM _overlaps, values
-  WHERE values.%I = _overlaps.%I', geom_geoid_colname, geom_geoid_colname);
+  WHERE values.%I = _overlaps.%I', data_geoid_colname, geom_geoid_colname);
 
   EXECUTE
     q

src/pg/sql/42_observatory_exploration.sql

@@ -114,7 +114,7 @@ BEGIN
       AND
         observatory.OBS_column.type = 'Geometry'
       AND
-        $1 && bounds::box2d
+        ST_Intersects($1, observatory.obs_table.the_geom)
   $string$ || timespan_query
   USING geom;
   RETURN;

src/pg/test/expected/40_observatory_utility_test.out

@@ -27,3 +27,6 @@ t
 _obs_standardizemeasurename_test
 t
 (1 row)
+obs_dumpversion_notnull
+t
+(1 row)

src/pg/test/expected/41_observatory_augmentation_test.out

@@ -76,3 +76,15 @@ t
 obs_getuscensuscategory_polygon
 t
 (1 row)
+obs_getmeasurebyid_cartodb_census_tract
+t
+(1 row)
+obs_getmeasurebyid_null_boundary_null_timespan
+t
+(1 row)
+obs_getmeasurebyid_cartodb_block_group
+t
+(1 row)
+obs_getmeasurebyid_nulls
+t
+(1 row)

src/pg/test/fixtures/drop_fixtures.sql

@@ -6,6 +6,7 @@ DROP TABLE IF EXISTS observatory.obs_column;
 DROP TABLE IF EXISTS observatory.obs_column_tag;
 DROP TABLE IF EXISTS observatory.obs_tag;
 DROP TABLE IF EXISTS observatory.obs_column_to_column;
+DROP TABLE IF EXISTS observatory.obs_dump_version;
 DROP TABLE IF EXISTS observatory.obs_65f29658e096ca1485bf683f65fdbc9f05ec3c5d;
 DROP TABLE IF EXISTS observatory.obs_1746e37b7cd28cb131971ea4187d42d71f09c5f3;
 DROP TABLE IF EXISTS observatory.obs_1a098da56badf5f32e336002b0a81708c40d29cd;

src/pg/test/sql/40_observatory_utility_test.sql

@@ -75,4 +75,7 @@ SELECT cdb_observatory._OBS_GetRelatedColumn(
 -- should give back a standardized measure name
 SELECT cdb_observatory._OBS_StandardizeMeasureName('test 343 %% 2 qqq }}{{}}') = 'test_343_2_qqq' As _OBS_StandardizeMeasureName_test;
 
+SELECT cdb_observatory.OBS_DumpVersion()
+  IS NOT NULL AS OBS_DumpVersion_notnull;
+
 \i test/fixtures/drop_fixtures.sql

src/pg/test/sql/41_observatory_augmentation_test.sql

@@ -204,4 +204,37 @@ SELECT cdb_observatory.OBS_GetUSCensusCategory(
   cdb_observatory._testarea(), 'Spielman-Singleton Segments: 10 Clusters') = 'Low income, mix of minorities' As OBS_GetUSCensusCategory_polygon;
 
 
+-- OBS_GetMeasureById tests
+-- typical query
+SELECT (cdb_observatory.OBS_GetMeasureById(
+  '36047048500',
+  'us.census.acs.B01003001',
+  'us.census.tiger.census_tract',
+  '2010 - 2014'
+) - 3241) / 3241 < 0.0001 As OBS_GetMeasureById_cartodb_census_tract;
+
+-- no boundary_id should give null
+SELECT cdb_observatory.OBS_GetMeasureById(
+  '36047048500',
+  'us.census.acs.B01003001',
+  NULL,
+  NULL
+) IS NULL As OBS_GetMeasureById_null_boundary_null_timespan;
+
+-- query at block_group level
+SELECT (cdb_observatory.OBS_GetMeasureById(
+  '360470485002',
+  'us.census.acs.B01003001',
+  'us.census.tiger.block_group',
+  '2010 - 2014'
+) - 1900) / 1900 < 0.0001 As OBS_GetMeasureById_cartodb_block_group;
+
+-- geom ref / boundary mismatch
+SELECT cdb_observatory.OBS_GetMeasureById(
+  '36047048500',
+  'us.census.acs.B01003001',
+  'us.census.tiger.block_group',
+  '2010 - 2014'
+) IS NULL As OBS_GetMeasureById_nulls;
+
 \i test/fixtures/drop_fixtures.sql