Release 1.0.2 artifact

Release 1.0.2 wit mocks for augment functions

NEWS.md

@@ -1,3 +1,30 @@
+1.0.2 (2016-07-12)
+---
+
+__Bugfixes__
+
+* Fix for `OBS_GetCategory` outside the US ([#135](https://github.com/CartoDB/observatory-extension/pull/137))
+* `OBS_GetMeasure` now respects the `normalize` parameter even when passed
+  a multi/polygon. Previously, no normalization was erroneously assumed.
+
+__Improvements__
+
+* Automated tests cover Mexico data
+* `obs_meta` is now provisioned during unit tests
+* `obs_meta` is now used during end-to-end tests
+* `OBS_GetMeasureByID` uses `obs_meta` internally, which should help
+  performance
+* `OBS_GetCategory` uses `obs_meta` internally, which should help perfromance
+* `OBS_GetCategory` will pick the correct category for an arbitrary polygon
+  (the category covering the highest % of that polygon)
+* `OBS_GetMeasure` has been updated to use `obs_meta` internally, which should
+  help performance
+* `OBS_GetMeasure` now can be passed "none" and skip normalization by area or
+  denominator for points
+* Fixtures are only loaded at the start of the unit test suite, and dropped at the end,
+  instead of at the start/end of each individual test file
+* Comment noisy NOTICEs ([#73](https://github.com/CartoDB/observatory-extension/issues/73))
+
 1.0.1 (2016-07-01)
 ---
 

doc/boundary_functions.md

@@ -1,8 +1,8 @@
 # Boundary Functions
 
-Use the following functions to retrieve [Boundary](/cartodb-platform/data/overview/#boundary-data) data. Data ranges from small areas (e.g. US Census Block Groups) to large areas (e.g. Countries). You can access boundaries by point location lookup, bounding box lookup, direct ID access and several other methods described below.
+Use the following functions to retrieve [Boundary](/carto-engine/data/overview/#boundary-data) data. Data ranges from small areas (e.g. US Census Block Groups) to large areas (e.g. Countries). You can access boundaries by point location lookup, bounding box lookup, direct ID access and several other methods described below.
 
-You can [access](/cartodb-platform/data/accessing/#accessing-the-data-observatory) boundaries through the CartoDB Editor. The same methods will work if you are using the CartoDB Platform to develop your application. We [encourage you](/cartodb-platform/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
+You can [access](/carto-engine/data/accessing/#accessing-the-data-observatory) boundaries through the CARTO Editor. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
 
 ## OBS_GetBoundariesByGeometry(polygon geometry, geometry_id text)
 

doc/discovery_functions.md

@@ -1,6 +1,6 @@
 # Discovery Functions
 
-If you are using the [discovery methods](/cartodb-platform/data/overview/#discovery-methods) from the Data Observatory, use the following functions to retrieve [boundary](/cartodb-platform/data/overview/#boundary-data) and [measures](/cartodb-platform/data/overview/#measures-data) data.
+If you are using the [discovery methods](/carto-engine/data/overview/#discovery-methods) from the Data Observatory, use the following functions to retrieve [boundary](/carto-engine/data/overview/#boundary-data) and [measures](/carto-engine/data/overview/#measures-data) data.
 
 ## OBS_Search(search_term)
 
@@ -47,7 +47,7 @@ A TABLE containing the following properties
 
 Key | Description
 --- | ---
-boundary_id | a boundary identifier from the [boundary ID glossary](/cartodb-platform/data/glossary/#boundary-ids)
+boundary_id | a boundary identifier from the [boundary ID glossary](/carto-engine/data/glossary/#boundary-ids)
 description | a brief description of the boundary dataset
 time_span | the timespan attached the boundary. this does not mean that the boundary is invalid outside of the timespan, but is the explicit timespan published with the geometry.
 

doc/measures_functions.md

@@ -1,10 +1,10 @@
 # Measures Functions
 
-[Data Observatory Measures](/cartodb-platform/data/overview/#measures-methods) are the numerical location data you can access. The measure functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (point or polygon) and receiving back a measure (an absolute value) for that location.
+[Data Observatory Measures](/carto-engine/data/overview/#measures-methods) are the numerical location data you can access. The measure functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (point or polygon) and receiving back a measure (an absolute value) for that location.
 
 There are hundreds of measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf).
 
-You can [access](/cartodb-platform/data/accessing/#accessing-the-data-observatory) measures through the CartoDB Editor. The same methods will work if you are using the CartoDB Platform to develop your application. We [encourage you](/cartodb-platform/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
+You can [access](/carto-engine/data/accessing/#accessing-the-data-observatory) measures through the CARTO Editor. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
 
 ## OBS_GetUSCensusMeasure(point geometry, measure_name text)
 

release/observatory.control

@@ -1,5 +1,5 @@
 comment = 'CartoDB Observatory backend extension'
-default_version = '1.0.1'
+default_version = '1.0.2'
 requires = 'postgis'
 superuser = true
 schema = cdb_observatory

scripts/autotest.py

@@ -24,17 +24,18 @@ def query(q, is_meta=False, **options):
     params['api_key'] = META_API_KEY if is_meta else API_KEY
     return requests.get(url, params=params)
 
-MEASURE_COLUMNS = [(r['id'], r['point_only'], ) for r in query('''
-SELECT id, aggregate NOT ILIKE 'sum' as point_only
-FROM obs_column
-WHERE type ILIKE 'numeric'
-AND weight > 0
+MEASURE_COLUMNS = [(r['numer_id'], r['point_only'], ) for r in query('''
+SELECT distinct numer_id, numer_aggregate NOT ILIKE 'sum' as point_only
+FROM obs_meta
+WHERE numer_type ILIKE 'numeric'
+AND numer_weight > 0
 ''', is_meta=True).json()['rows']]
 
-CATEGORY_COLUMNS = [(r['id'], ) for r in query('''
-SELECT id FROM obs_column
-WHERE type ILIKE 'text'
-AND weight > 0
+CATEGORY_COLUMNS = [(r['numer_id'], ) for r in query('''
+SELECT distinct numer_id
+FROM obs_meta
+WHERE numer_type ILIKE 'text'
+AND numer_weight > 0
 ''', is_meta=True).json()['rows']]
 
 BOUNDARY_COLUMNS = [(r['id'], ) for r in query('''
@@ -43,12 +44,12 @@ WHERE type ILIKE 'geometry'
 AND weight > 0
 ''', is_meta=True).json()['rows']]
 
-US_CENSUS_MEASURE_COLUMNS = [(r['name'], ) for r in query('''
-SELECT c.name FROM obs_column c, obs_column_tag ct
-WHERE type ILIKE 'numeric'
-AND c.id = ct.column_id
-AND ct.tag_id LIKE 'us.census%'
-AND weight > 0
+US_CENSUS_MEASURE_COLUMNS = [(r['numer_name'], ) for r in query('''
+SELECT distinct numer_name
+FROM obs_meta
+WHERE numer_type ILIKE 'numeric'
+AND 'us.census.acs.acs' = ANY (subsection_tags)
+AND numer_weight > 0
 ''', is_meta=True).json()['rows']]
 
 
@@ -95,6 +96,8 @@ def default_point(column_id):
         return 'CDB_LatLng(42.8226119029222, -2.51141249535454)'
     elif column_id.startswith('us.zillow'):
         return 'CDB_LatLng(28.3305906291771, -81.3544048197256)'
+    elif column_id.startswith('mx.'):
+        return 'CDB_LatLng(19.41347699386547, -99.17019367218018)'
     else:
         return 'CDB_LatLng(40.7, -73.9)'
 
@@ -110,6 +113,7 @@ def default_area(column_id):
 
 @parameterized(US_CENSUS_MEASURE_COLUMNS)
 def test_get_us_census_measure_points(name):
+    print 'test_get_us_census_measure_points, ', name
     resp = query('''
 SELECT * FROM {schema}OBS_GetUSCensusMeasure({point}, '{name}')
                  '''.format(name=name.replace("'", "''"),
@@ -123,6 +127,7 @@ SELECT * FROM {schema}OBS_GetUSCensusMeasure({point}, '{name}')
 
 @parameterized(MEASURE_COLUMNS)
 def test_get_measure_areas(column_id, point_only):
+    print 'test_get_measure_areas, ', column_id, point_only
     if point_only:
         return
     resp = query('''
@@ -138,6 +143,7 @@ SELECT * FROM {schema}OBS_GetMeasure({area}, '{column_id}')
 
 @parameterized(MEASURE_COLUMNS)
 def test_get_measure_points(column_id, point_only):
+    print 'test_get_measure_points, ', column_id, point_only
     resp = query('''
 SELECT * FROM {schema}OBS_GetMeasure({point}, '{column_id}')
                  '''.format(column_id=column_id,
@@ -162,6 +168,7 @@ SELECT * FROM {schema}OBS_GetMeasure({point}, '{column_id}')
 
 @parameterized(CATEGORY_COLUMNS)
 def test_get_category_points(column_id):
+    print 'test_get_category_points, ', column_id
     resp = query('''
 SELECT * FROM {schema}OBS_GetCategory({point}, '{column_id}')
                  '''.format(column_id=column_id,

scripts/generate_fixtures.py

@@ -34,7 +34,7 @@ def select_star(tablename):
 cdb = Dumpr('observatory.cartodb.com','')
 
 metadata = ['obs_table', 'obs_column_table', 'obs_column', 'obs_column_tag',
-            'obs_tag', 'obs_column_to_column', 'obs_dump_version']
+            'obs_tag', 'obs_column_to_column', 'obs_dump_version', ]
 
 fixtures = [
     ('us.census.tiger.census_tract', 'us.census.tiger.census_tract', '2014'),
@@ -90,3 +90,124 @@ with open('src/pg/test/fixtures/load_fixtures.sql', 'w') as outfile:
             cdb.dump(' '.join([select_star(tablename), "WHERE {}::text {} {}".format(colname, compare, where)]),
                      tablename, outfile, schema='observatory')
             dropfiles.write('DROP TABLE IF EXISTS observatory.{};\n'.format(tablename))
+
+
+        outfile.write('''
+ALTER TABLE observatory.obs_table
+  ADD PRIMARY KEY (id);
+ALTER TABLE observatory.obs_column_table
+  ADD PRIMARY KEY (column_id, table_id);
+CREATE UNIQUE INDEX ON observatory.obs_column_table (table_id, column_id);
+CREATE UNIQUE INDEX ON observatory.obs_column_table (table_id, colname);
+ALTER TABLE observatory.obs_column
+  ADD PRIMARY KEY (id);
+ALTER TABLE observatory.obs_column_to_column
+  ADD PRIMARY KEY (source_id, target_id, reltype);
+CREATE UNIQUE INDEX ON observatory.obs_column_to_column (target_id, source_id, reltype);
+CREATE INDEX ON observatory.obs_column_to_column (reltype);
+ALTER TABLE observatory.obs_column_tag
+  ADD PRIMARY KEY (column_id, tag_id);
+CREATE UNIQUE INDEX ON observatory.obs_column_tag (tag_id, column_id);
+ALTER TABLE observatory.obs_tag
+  ADD PRIMARY KEY (id);
+CREATE INDEX ON observatory.obs_tag (type);
+
+VACUUM ANALYZE observatory.obs_table;
+VACUUM ANALYZE observatory.obs_column_table;
+VACUUM ANALYZE observatory.obs_column;
+VACUUM ANALYZE observatory.obs_column_to_column;
+VACUUM ANALYZE observatory.obs_column_tag;
+VACUUM ANALYZE observatory.obs_tag;
+
+CREATE TABLE observatory.obs_meta AS
+SELECT numer_c.id numer_id,
+       denom_c.id denom_id,
+       geom_c.id geom_id,
+       MAX(numer_c.name) numer_name,
+       MAX(denom_c.name) denom_name,
+       MAX(geom_c.name) geom_name,
+       MAX(numer_c.description) numer_description,
+       MAX(denom_c.description) denom_description,
+       MAX(geom_c.description) geom_description,
+       MAX(numer_c.aggregate) numer_aggregate,
+       MAX(denom_c.aggregate) denom_aggregate,
+       MAX(geom_c.aggregate) geom_aggregate,
+       MAX(numer_c.type) numer_type,
+       MAX(denom_c.type) denom_type,
+       MAX(geom_c.type) geom_type,
+       MAX(numer_data_ct.colname) numer_colname,
+       MAX(denom_data_ct.colname) denom_colname,
+       MAX(geom_geom_ct.colname) geom_colname,
+       MAX(numer_geomref_ct.colname) numer_geomref_colname,
+       MAX(denom_geomref_ct.colname) denom_geomref_colname,
+       MAX(geom_geomref_ct.colname) geom_geomref_colname,
+       MAX(numer_t.tablename) numer_tablename,
+       MAX(denom_t.tablename) denom_tablename,
+       MAX(geom_t.tablename) geom_tablename,
+       MAX(numer_t.timespan) numer_timespan,
+       MAX(denom_t.timespan) denom_timespan,
+       MAX(numer_c.weight) numer_weight,
+       MAX(denom_c.weight) denom_weight,
+       MAX(geom_c.weight) geom_weight,
+       MAX(geom_t.timespan) geom_timespan,
+       MAX(geom_t.the_geom_webmercator)::geometry AS the_geom_webmercator,
+       ARRAY_AGG(DISTINCT s_tag.id) section_tags,
+       ARRAY_AGG(DISTINCT ss_tag.id) subsection_tags,
+       ARRAY_AGG(DISTINCT unit_tag.id) unit_tags
+FROM observatory.obs_column_table numer_data_ct,
+     observatory.obs_table numer_t,
+     observatory.obs_column_table numer_geomref_ct,
+     observatory.obs_column geomref_c,
+     observatory.obs_column_to_column geomref_c2c,
+     observatory.obs_column geom_c,
+     observatory.obs_column_table geom_geom_ct,
+     observatory.obs_column_table geom_geomref_ct,
+     observatory.obs_table geom_t,
+     observatory.obs_column_tag ss_ctag,
+     observatory.obs_tag ss_tag,
+     observatory.obs_column_tag s_ctag,
+     observatory.obs_tag s_tag,
+     observatory.obs_column_tag unit_ctag,
+     observatory.obs_tag unit_tag,
+     observatory.obs_column numer_c
+  LEFT JOIN (
+    observatory.obs_column_to_column denom_c2c
+    JOIN observatory.obs_column denom_c ON denom_c2c.target_id = denom_c.id
+    JOIN observatory.obs_column_table denom_data_ct ON denom_data_ct.column_id = denom_c.id
+    JOIN observatory.obs_table denom_t ON denom_data_ct.table_id = denom_t.id
+    JOIN observatory.obs_column_table denom_geomref_ct ON denom_geomref_ct.table_id = denom_t.id
+  ) ON denom_c2c.source_id = numer_c.id
+WHERE numer_c.id = numer_data_ct.column_id
+  AND numer_data_ct.table_id = numer_t.id
+  AND numer_t.id = numer_geomref_ct.table_id
+  AND numer_geomref_ct.column_id = geomref_c.id
+  AND geomref_c2c.reltype = 'geom_ref'
+  AND geomref_c.id = geomref_c2c.source_id
+  AND geom_c.id = geomref_c2c.target_id
+  AND geom_geomref_ct.column_id = geomref_c.id
+  AND geom_geomref_ct.table_id = geom_t.id
+  AND geom_geom_ct.column_id = geom_c.id
+  AND geom_geom_ct.table_id = geom_t.id
+  AND geom_c.type ILIKE 'geometry'
+  AND numer_c.type NOT ILIKE 'geometry'
+  AND numer_t.id != geom_t.id
+  AND numer_c.id != geomref_c.id
+  AND unit_tag.type = 'unit'
+  AND ss_tag.type = 'subsection'
+  AND s_tag.type = 'section'
+  AND unit_ctag.column_id = numer_c.id
+  AND unit_ctag.tag_id = unit_tag.id
+  AND ss_ctag.column_id = numer_c.id
+  AND ss_ctag.tag_id = ss_tag.id
+  AND s_ctag.column_id = numer_c.id
+  AND s_ctag.tag_id = s_tag.id
+  AND (denom_c2c.reltype = 'denominator' OR denom_c2c.reltype IS NULL)
+  AND (denom_geomref_ct.column_id = geomref_c.id OR denom_geomref_ct.column_id IS NULL)
+  AND (denom_t.timespan = numer_t.timespan OR denom_t.timespan IS NULL)
+GROUP BY numer_c.id, denom_c.id, geom_c.id,
+         numer_t.id, denom_t.id, geom_t.id;
+        ''')
+
+        dropfiles.write('''
+DROP TABLE IF EXISTS observatory.obs_meta;
+                        ''')

src/pg/observatory.control

@@ -1,5 +1,5 @@
 comment = 'CartoDB Observatory backend extension'
-default_version = '1.0.1'
+default_version = '1.0.2'
 requires = 'postgis'
 superuser = true
 schema = cdb_observatory

src/pg/sql/15_fdw_utilities.sql

@@ -0,0 +1,67 @@
+CREATE OR REPLACE FUNCTION cdb_observatory._OBS_ConnectRemoteTable(fdw_name text, schema_name text, user_dbname text, user_hostname text, username text, user_tablename text, user_schema text)
+RETURNS void
+AS $$
+DECLARE
+  row record;
+  option record;
+  connection_str json;
+BEGIN
+  -- Build connection string
+  connection_str := '{"server":{"extensions":"postgis", "dbname":"'
+    || user_dbname ||'", "host":"' || user_hostname ||'", "port":"5432"}, "users":{"public"'
+    || ':{"user":"' || username ||'", "password":""} } }';
+
+  -- This function tries to be as idempotent as possible, by not creating anything more than once
+  -- (not even using IF NOT EXIST to avoid throwing warnings)
+  IF NOT EXISTS ( SELECT * FROM pg_extension WHERE extname = 'postgres_fdw') THEN
+    CREATE EXTENSION postgres_fdw;
+  END IF;
+  -- Create FDW first if it does not exist
+  IF NOT EXISTS ( SELECT * FROM pg_foreign_server WHERE srvname = fdw_name)
+    THEN
+    EXECUTE FORMAT('CREATE SERVER %I FOREIGN DATA WRAPPER postgres_fdw', fdw_name);
+  END IF;
+
+  -- Set FDW settings
+  FOR row IN SELECT p.key, p.value from lateral json_each_text(connection_str->'server') p
+    LOOP
+      IF NOT EXISTS (WITH a AS (select split_part(unnest(srvoptions), '=', 1) as options from pg_foreign_server where srvname=fdw_name) SELECT * from a where options = row.key)
+        THEN
+        EXECUTE FORMAT('ALTER SERVER %I OPTIONS (ADD %I %L)', fdw_name, row.key, row.value);
+      ELSE
+        EXECUTE FORMAT('ALTER SERVER %I OPTIONS (SET %I %L)', fdw_name, row.key, row.value);
+      END IF;
+    END LOOP;
+
+    -- Create user mappings
+    FOR row IN SELECT p.key, p.value from lateral json_each(connection_str->'users') p LOOP
+      -- Check if entry on pg_user_mappings exists
+      IF NOT EXISTS ( SELECT * FROM pg_user_mappings WHERE srvname = fdw_name AND usename = row.key ) THEN
+        EXECUTE FORMAT ('CREATE USER MAPPING FOR %I SERVER %I', row.key, fdw_name);
+      END IF;
+
+      -- Update user mapping settings
+      FOR option IN SELECT o.key, o.value from lateral json_each_text(row.value) o LOOP
+        IF NOT EXISTS (WITH a AS (select split_part(unnest(umoptions), '=', 1) as options from pg_user_mappings WHERE srvname = fdw_name AND usename = row.key) SELECT * from a where options = option.key) THEN
+          EXECUTE FORMAT('ALTER USER MAPPING FOR %I SERVER %I OPTIONS (ADD %I %L)', row.key, fdw_name, option.key, option.value);
+        ELSE
+          EXECUTE FORMAT('ALTER USER MAPPING FOR %I SERVER %I OPTIONS (SET %I %L)', row.key, fdw_name, option.key, option.value);
+        END IF;
+      END LOOP;
+    END LOOP;
+
+    -- Create schema if it does not exist.
+    IF NOT EXISTS ( SELECT * from pg_namespace WHERE nspname=fdw_name) THEN
+      EXECUTE FORMAT ('CREATE SCHEMA %I', fdw_name);
+    END IF;
+
+    -- Bring the remote cdb_tablemetadata
+    IF NOT EXISTS ( SELECT * FROM PG_CLASS WHERE relnamespace = (SELECT oid FROM pg_namespace WHERE nspname=fdw_name) and relname='cdb_tablemetadata') THEN
+      EXECUTE FORMAT ('CREATE FOREIGN TABLE %I.cdb_tablemetadata (tabname text, updated_at timestamp with time zone) SERVER %I OPTIONS (table_name ''cdb_tablemetadata_text'', schema_name ''public'', updatable ''false'')', fdw_name, fdw_name);
+    END IF;
+
+    -- Import target table
+    EXECUTE FORMAT ('IMPORT FOREIGN SCHEMA %I LIMIT TO (%I) from SERVER %I INTO %I', user_schema, user_tablename, fdw_name, schema_name);
+
+END;
+$$ LANGUAGE PLPGSQL;

src/pg/sql/40_observatory_utility.sql

@@ -158,28 +158,6 @@ BEGIN
 END;
 $$ LANGUAGE plpgsql;
 
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetRelatedColumn(columns_ids text[], reltype text )
-RETURNS TEXT[]
-AS $$
-DECLARE
-  result TEXT[];
-BEGIN
-  EXECUTE '
-    With ids as (
-      select row_number() over() as no, id from (select unnest($1) as id) t
-    )
-    select array_agg(target_id order by no)
-    FROM  ids
-    LEFT JOIN observatory.obs_column_to_column
-    on  source_id  = id
-    where reltype = $2 or reltype is null
-  '
-  INTO result
-  using columns_ids, reltype;
-  return result;
-END;
-$$ LANGUAGE plpgsql;
-
 -- Function that replaces all non digits or letters with _ trims and lowercases the
 -- passed measure name
 

src/pg/sql/41_observatory_augmentation.sql

@@ -169,19 +169,19 @@ BEGIN
 
   IF geom_table_name IS NULL
   THEN
-     RAISE NOTICE 'Point % is outside of the data region', ST_AsText(geom);
+     --raise notice 'Point % is outside of the data region', ST_AsText(geom);
       -- TODO this should return JSON
      RETURN QUERY SELECT '{}'::json;
      RETURN;
   END IF;
 
   IF data_table_info IS NULL THEN
-    RAISE NOTICE 'Cannot find data table for boundary ID %, column_ids %, and time_span %', geometry_level, column_ids, time_span;
+    --raise notice 'Cannot find data table for boundary ID %, column_ids %, and time_span %', geometry_level, column_ids, time_span;
   END IF;
 
   IF ST_GeometryType(geom) = 'ST_Point'
   THEN
-    RAISE NOTICE 'geom_table_name %, data_table_info %', geom_table_name, data_table_info::json[];
+    --raise notice 'geom_table_name %, data_table_info %', geom_table_name, data_table_info::json[];
     results := cdb_observatory._OBS_GetPoints(geom,
                                               geom_table_name,
                                               data_table_info);
@@ -260,7 +260,7 @@ BEGIN
   USING geom
   INTO geoid;
 
-  RAISE NOTICE 'geoid is %, geometry table is % ', geoid, geom_table_name;
+  --raise notice 'geoid is %, geometry table is % ', geoid, geom_table_name;
 
   EXECUTE
     format('SELECT ST_Area(the_geom::geography) / (1000 * 1000)
@@ -273,7 +273,7 @@ BEGIN
 
   IF area IS NULL
   THEN
-    RAISE NOTICE 'No geometry at %', ST_AsText(geom);
+    --raise notice 'No geometry at %', ST_AsText(geom);
   END IF;
 
   query := 'SELECT Array[';
@@ -338,44 +338,173 @@ $$ LANGUAGE plpgsql;
 CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetMeasure(
   geom geometry(Geometry, 4326),
   measure_id TEXT,
-  normalize TEXT DEFAULT 'area', -- TODO none/null
+  normalize TEXT DEFAULT NULL,
   boundary_id TEXT DEFAULT NULL,
   time_span TEXT DEFAULT NULL
 )
 RETURNS NUMERIC
 AS $$
 DECLARE
+  geom_type TEXT;
+  map_type TEXT;
+  numer_aggregate TEXT;
+  numer_colname TEXT;
+  numer_geomref_colname TEXT;
+  numer_tablename TEXT;
+  denom_colname TEXT;
+  denom_geomref_colname TEXT;
+  denom_tablename TEXT;
+  geom_colname TEXT;
+  geom_geomref_colname TEXT;
+  geom_tablename TEXT;
   result NUMERIC;
-  measure_ids TEXT[];
-  denominator_id TEXT;
-  vals NUMERIC[];
+  sql TEXT;
+  numer_name TEXT;
 BEGIN
 
-  IF normalize ILIKE 'area' THEN
-    measure_ids := ARRAY[measure_id];
+  EXECUTE
+     $query$
+     SELECT numer_aggregate, numer_colname, numer_geomref_colname, numer_tablename,
+            denom_colname, denom_geomref_colname, denom_tablename,
+            geom_colname, geom_geomref_colname, geom_tablename, numer_name
+             FROM observatory.obs_meta
+             WHERE (geom_id = $1 OR ($1 = ''))
+               AND numer_id = $2
+               AND (numer_timespan = $3 OR ($3 = ''))
+             ORDER BY geom_weight DESC, numer_timespan DESC
+             LIMIT 1
+     $query$
+    INTO numer_aggregate, numer_colname, numer_geomref_colname, numer_tablename,
+         denom_colname, denom_geomref_colname, denom_tablename,
+         geom_colname, geom_geomref_colname, geom_tablename, numer_name
+    USING COALESCE(boundary_id, ''), measure_id, COALESCE(time_span, '');
+
+  IF ST_GeometryType(geom) = 'ST_Point' THEN
+    geom_type := 'point';
+  ELSIF ST_GeometryType(geom) IN ('ST_Polygon', 'ST_MultiPolygon') THEN
+    geom_type := 'polygon';
+  ELSE
+    RAISE EXCEPTION 'Invalid geometry type (%), can only handle ''ST_Point'', ''ST_Polygon'', and ''ST_MultiPolygon''',
+                    ST_GeometryType(geom);
+  END IF;
+
+  IF normalize ILIKE 'area' AND numer_aggregate ILIKE 'sum' THEN
+    map_type := 'areaNormalized';
   ELSIF normalize ILIKE 'denominator' THEN
-    EXECUTE 'SELECT (cdb_observatory._OBS_GetRelatedColumn(ARRAY[$1], ''denominator''))[1]
-    ' INTO denominator_id
-    USING measure_id;
-    measure_ids := ARRAY[measure_id, denominator_id];
-  ELSIF normalize ILIKE 'none' THEN
-    -- TODO we need a switch on obs_get to disable area normalization
-    RAISE EXCEPTION 'No normalization not yet supported.';
+    map_type := 'denominated';
   ELSE
-    RAISE EXCEPTION 'Only valid inputs for "normalize" are "area" (default) and "denominator".';
+    -- defaults: area normalization for point if it's possible and none for
+    -- polygon or non-summable point
+    IF geom_type = 'point' AND numer_aggregate ILIKE 'sum' THEN
+      map_type := 'areaNormalized';
+    ELSE
+      map_type := 'predenominated';
+    END IF;
   END IF;
 
-  EXECUTE '
-    SELECT ARRAY_AGG(val) FROM (SELECT (cdb_observatory._OBS_Get($1, $2, $3, $4)->>''value'')::NUMERIC val) b
-  '
-  INTO vals
-  USING geom, measure_ids, time_span, boundary_id;
-
-  IF normalize ILIKE 'denominator' THEN
-    RETURN (vals)[1]/(vals)[2];
-  ELSE
-    RETURN (vals)[1];
+  IF geom_type = 'point' THEN
+    IF map_type = 'areaNormalized' THEN
+      sql = format('WITH _geom AS (SELECT ST_Area(geom.%I::Geography) / 1000000 area, geom.%I geom_ref
+                                   FROM observatory.%I geom
+                                   WHERE ST_Within(%L, geom.%I)
+                                   LIMIT 1)
+                    SELECT numer.%I / (SELECT area FROM _geom)
+                    FROM observatory.%I numer
+                    WHERE numer.%I = (SELECT geom_ref FROM _geom)',
+                geom_colname, geom_geomref_colname, geom_tablename,
+                geom, geom_colname, numer_colname, numer_tablename,
+                numer_geomref_colname);
+    ELSIF map_type = 'denominated' THEN
+      sql = format('SELECT numer.%I / NULLIF((SELECT denom.%I FROM observatory.%I denom WHERE denom.%I = numer.%I LIMIT 1), 0)
+                    FROM observatory.%I numer
+                    WHERE numer.%I = (SELECT geom.%I FROM observatory.%I geom WHERE ST_Within(%L, geom.%I) LIMIT 1)',
+                        numer_colname, denom_colname, denom_tablename,
+                        denom_geomref_colname, numer_geomref_colname,
+                        numer_tablename,
+                        numer_geomref_colname, geom_geomref_colname,
+                        geom_tablename, geom, geom_colname);
+    ELSIF map_type = 'predenominated' THEN
+      sql = format('SELECT numer.%I
+                    FROM observatory.%I numer
+                    WHERE numer.%I = (SELECT geom.%I FROM observatory.%I geom WHERE ST_Within(%L, geom.%I) LIMIT 1)',
+                        numer_colname, numer_tablename,
+                        numer_geomref_colname, geom_geomref_colname, geom_tablename,
+                        geom, geom_colname);
+    END IF;
+  ELSIF geom_type = 'polygon' THEN
+    IF map_type = 'areaNormalized' THEN
+      sql = format('WITH _geom AS (SELECT ST_Area(ST_Intersection(%L, geom.%I))
+                                        / ST_Area(geom.%I) overlap, geom.%I geom_ref
+                                   FROM observatory.%I geom
+                                   WHERE ST_Intersects(%L, geom.%I)
+                                     AND ST_Area(ST_Intersection(%L, geom.%I)) / ST_Area(geom.%I) > 0)
+                    SELECT SUM(numer.%I * (SELECT _geom.overlap FROM _geom WHERE _geom.geom_ref = numer.%I)) /
+                           ST_Area(%L::Geography)
+                    FROM observatory.%I numer
+                    WHERE numer.%I = ANY ((SELECT ARRAY_AGG(geom_ref) FROM _geom)::TEXT[])',
+                geom, geom_colname, geom_colname,
+                geom_geomref_colname, geom_tablename,
+                geom, geom_colname,
+                geom, geom_colname, geom_colname,
+                numer_colname, numer_geomref_colname,
+                geom, numer_tablename,
+                numer_geomref_colname);
+    ELSIF map_type = 'denominated' THEN
+      sql = format('WITH _geom AS (SELECT ST_Area(ST_Intersection(%L, geom.%I))
+                                        / ST_Area(geom.%I) overlap, geom.%I geom_ref
+                                   FROM observatory.%I geom
+                                   WHERE ST_Intersects(%L, geom.%I)
+                                     AND ST_Area(ST_Intersection(%L, geom.%I)) / ST_Area(geom.%I) > 0),
+                        _denom AS (SELECT denom.%I, denom.%I geom_ref
+                                   FROM observatory.%I denom
+                                   WHERE denom.%I = ANY ((SELECT ARRAY_AGG(geom_ref) FROM _geom)::TEXT[]))
+                    SELECT SUM(numer.%I * (SELECT _geom.overlap FROM _geom WHERE _geom.geom_ref = numer.%I)) /
+                           SUM((SELECT _denom.%I * (SELECT _geom.overlap
+                                                    FROM _geom
+                                                    WHERE _geom.geom_ref = _denom.geom_ref)
+                                FROM _denom WHERE _denom.geom_ref = numer.%I))
+                    FROM observatory.%I numer
+                    WHERE numer.%I = ANY ((SELECT ARRAY_AGG(geom_ref) FROM _geom)::TEXT[])',
+                geom, geom_colname,
+                geom_colname, geom_geomref_colname,
+                geom_tablename,
+                geom, geom_colname,
+                geom, geom_colname, geom_colname,
+                denom_colname, denom_geomref_colname,
+                denom_tablename,
+                denom_geomref_colname,
+                numer_colname, numer_geomref_colname,
+                denom_colname,
+                numer_geomref_colname,
+                numer_tablename,
+                numer_geomref_colname);
+    ELSIF map_type = 'predenominated' THEN
+      IF numer_aggregate NOT ILIKE 'sum' THEN
+        RAISE EXCEPTION 'Cannot calculate "%" (%) for custom area as it cannot be summed, use ST_PointOnSurface instead',
+                        numer_name, numer_id;
+      ELSE
+        sql = format('WITH _geom AS (SELECT ST_Area(ST_Intersection(%L, geom.%I))
+                                          / ST_Area(geom.%I) overlap, geom.%I geom_ref
+                                     FROM observatory.%I geom
+                                     WHERE ST_Intersects(%L, geom.%I)
+                                       AND ST_Area(ST_Intersection(%L, geom.%I)) / ST_Area(geom.%I) > 0)
+                      SELECT SUM(numer.%I * (SELECT _geom.overlap FROM _geom WHERE _geom.geom_ref = numer.%I))
+                      FROM observatory.%I numer
+                      WHERE numer.%I = ANY ((SELECT ARRAY_AGG(geom_ref) FROM _geom)::TEXT[])',
+                  geom, geom_colname, geom_colname,
+                  geom_geomref_colname, geom_tablename,
+                  geom, geom_colname,
+                  geom, geom_colname, geom_colname,
+                  numer_colname, numer_geomref_colname,
+                  numer_tablename,
+                  numer_geomref_colname);
+      END IF;
+    END IF;
   END IF;
+
+  EXECUTE sql INTO result;
+  RETURN result;
+
 END;
 $$ LANGUAGE plpgsql;
 
@@ -393,33 +522,30 @@ DECLARE
   colname TEXT;
   measure_val NUMERIC;
   data_geoid_colname TEXT;
-  test_query TEXT;
 BEGIN
 
-  SELECT x ->> 'colname', x ->> 'tablename' INTO colname, target_table
-  FROM cdb_observatory._OBS_GetColumnData(boundary_id, Array[measure_id], time_span) As x;
-
   EXECUTE
-    format('SELECT ct.colname
-              FROM observatory.obs_column_to_column c2c,
-                   observatory.obs_column_table ct,
-                   observatory.obs_table t
-             WHERE c2c.reltype = ''geom_ref''
-               AND ct.column_id = c2c.source_id
-               AND ct.table_id = t.id
-               AND t.tablename = %L'
-     , target_table)
-  INTO data_geoid_colname;
+     $query$
+     SELECT numer_colname, numer_geomref_colname, numer_tablename
+             FROM observatory.obs_meta
+             WHERE (geom_id = $1 OR ($1 = ''))
+               AND numer_id = $2
+               AND (numer_timespan = $3 OR ($3 = ''))
+             ORDER BY geom_weight DESC, numer_timespan DESC
+             LIMIT 1
+     $query$
+    INTO colname, data_geoid_colname, target_table
+    USING COALESCE(boundary_id, ''), measure_id, COALESCE(time_span, '');
 
-  RAISE DEBUG 'target_table %, colname %', target_table, colname;
+  --RAISE DEBUG 'target_table %, colname %', target_table, colname;
 
   EXECUTE format(
       'SELECT %I
-       FROM observatory.%I
-       WHERE %I.%I = %L',
+       FROM observatory.%I data
+       WHERE data.%I = %L',
        colname,
        target_table,
-       target_table, data_geoid_colname, geom_ref)
+       data_geoid_colname, geom_ref)
   INTO measure_val;
 
   RETURN measure_val;
@@ -436,27 +562,61 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetCategory(
 RETURNS TEXT
 AS $$
 DECLARE
-  denominator_id TEXT;
-  categories TEXT[];
+  data_table TEXT;
+  geom_table TEXT;
+  colname TEXT;
+  data_geomref_colname TEXT;
+  geom_geomref_colname TEXT;
+  geom_colname TEXT;
+  category_val TEXT;
+  category_share NUMERIC;
 BEGIN
 
-  IF boundary_id IS NULL THEN
-    -- TODO we should determine best boundary for this geom
-    boundary_id := 'us.census.tiger.census_tract';
+  EXECUTE
+     $query$
+     SELECT numer_colname, numer_geomref_colname, numer_tablename,
+            geom_geomref_colname, geom_colname, geom_tablename
+             FROM observatory.obs_meta
+             WHERE (geom_id = $1 OR ($1 = ''))
+               AND numer_id = $2
+               AND (numer_timespan = $3 OR ($3 = ''))
+             ORDER BY geom_weight DESC, numer_timespan DESC
+             LIMIT 1
+     $query$
+    INTO colname, data_geomref_colname, data_table,
+         geom_geomref_colname, geom_colname, geom_table
+    USING COALESCE(boundary_id, ''), category_id, COALESCE(time_span, '');
+
+  IF ST_GeometryType(geom) = 'ST_Point' THEN
+    EXECUTE format(
+        'SELECT data.%I
+         FROM observatory.%I data, observatory.%I geom
+         WHERE data.%I = geom.%I
+           AND ST_WITHIN(%L, geom.%I) ',
+         colname, data_table, geom_table, data_geomref_colname,
+         geom_geomref_colname, geom, geom_colname)
+    INTO category_val;
+  ELSE
+    -- favor the category with the most area
+    EXECUTE format(
+       'SELECT data.%I category, SUM(overlap_fraction) category_share
+        FROM observatory.%I data, (
+          SELECT ST_Area(
+           ST_Intersection(%L, a.%I)
+          ) / ST_Area(%L) AS overlap_fraction, a.%I geomref
+          FROM observatory.%I as a
+          WHERE %L && a.%I) _overlaps
+        WHERE data.%I = _overlaps.geomref
+        GROUP BY category
+        ORDER BY SUM(overlap_fraction) DESC
+        LIMIT 1',
+          colname, data_table,
+          geom, geom_colname, geom, geom_geomref_colname,
+          geom_table, geom, geom_colname, data_geomref_colname)
+    INTO category_val, category_share;
   END IF;
 
-  IF time_span IS NULL THEN
-    -- TODO we should determine latest timespan for this measure
-    time_span := '2010 - 2014';
-  END IF;
-
-  EXECUTE '
-    SELECT ARRAY_AGG(val) FROM (SELECT (cdb_observatory._OBS_GetCategories($1, $2, $3, $4))->>''category'' val LIMIT 1) b
-  '
-  INTO categories
-  USING geom, ARRAY[category_id], boundary_id, time_span;
-
-  RETURN (categories)[1];
+  RETURN category_val;
 
 END;
 $$ LANGUAGE plpgsql;
@@ -464,7 +624,7 @@ $$ LANGUAGE plpgsql;
 CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetUSCensusMeasure(
    geom geometry(Geometry, 4326),
    name TEXT,
-   normalize TEXT DEFAULT 'area',
+   normalize TEXT DEFAULT NULL,
    boundary_id TEXT DEFAULT NULL,
    time_span TEXT DEFAULT NULL
  )
@@ -530,7 +690,7 @@ $$ LANGUAGE plpgsql;
 
 CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetPopulation(
   geom geometry(Geometry, 4326),
-  normalize TEXT DEFAULT 'area',
+  normalize TEXT DEFAULT NULL,
   boundary_id TEXT DEFAULT NULL,
   time_span TEXT DEFAULT NULL
 )
@@ -796,7 +956,7 @@ BEGIN
 
   IF geom_table_name IS NULL
   THEN
-     RAISE NOTICE 'Point % is outside of the data region', ST_AsText(geom);
+     --raise notice 'Point % is outside of the data region', ST_AsText(geom);
      RETURN QUERY SELECT '{}'::text[], '{}'::text[];
      RETURN;
   END IF;
@@ -810,7 +970,7 @@ BEGIN
 
   IF data_table_info IS NULL
   THEN
-    RAISE NOTICE 'No data table found for this location';
+    --raise notice 'No data table found for this location';
     RETURN QUERY SELECT NULL::json;
     RETURN;
   END IF;
@@ -825,7 +985,7 @@ BEGIN
 
   IF geoid IS NULL
   THEN
-    RAISE NOTICE 'No geometry id for this location';
+    --raise notice 'No geometry id for this location';
     RETURN QUERY SELECT NULL::json;
     RETURN;
   END IF;

src/pg/sql/44_observatory_geometries.sql

@@ -64,11 +64,11 @@ BEGIN
   -- if no tables are found, raise notice and return null
   IF target_table IS NULL
   THEN
-    RAISE NOTICE 'No boundaries found for ''%'' in ''%''', ST_AsText(geom), boundary_id;
+    --RAISE NOTICE 'No boundaries found for ''%'' in ''%''', ST_AsText(geom), boundary_id;
     RETURN NULL::geometry;
   END IF;
 
-  RAISE NOTICE 'target_table: %', target_table;
+  --RAISE NOTICE 'target_table: %', target_table;
 
   -- return the first boundary in intersections
   EXECUTE format(
@@ -143,7 +143,7 @@ BEGIN
   -- if no tables are found, raise notice and return null
   IF target_table IS NULL
   THEN
-    RAISE NOTICE 'Warning: No boundaries found for ''%''', boundary_id;
+    --RAISE NOTICE 'Warning: No boundaries found for ''%''', boundary_id;
     RETURN NULL::text;
   END IF;
 
@@ -159,7 +159,7 @@ BEGIN
    , target_table)
   INTO geoid_colname;
 
-  RAISE NOTICE 'target_table: %, geoid_colname: %', target_table, geoid_colname;
+  --RAISE NOTICE 'target_table: %, geoid_colname: %', target_table, geoid_colname;
 
   -- return geometry id column value
   EXECUTE format(
@@ -212,11 +212,11 @@ BEGIN
   SELECT * INTO geoid_colname, target_table, geom_colname
   FROM cdb_observatory._OBS_GetGeometryMetadata(boundary_id);
 
-  RAISE NOTICE '%', target_table;
+  --RAISE NOTICE '%', target_table;
 
   IF target_table IS NULL
   THEN
-    RAISE NOTICE 'No geometries found';
+    --RAISE NOTICE 'No geometries found';
     RETURN NULL::geometry;
   END IF;
 
@@ -272,12 +272,12 @@ BEGIN
   -- if no tables are found, raise notice and return null
   IF target_table IS NULL
   THEN
-    RAISE NOTICE 'No boundaries found for bounding box ''%'' in ''%''', ST_AsText(geom), boundary_id;
+    --RAISE NOTICE 'No boundaries found for bounding box ''%'' in ''%''', ST_AsText(geom), boundary_id;
     RETURN QUERY SELECT NULL::geometry, NULL::text;
     RETURN;
   END IF;
 
-  RAISE NOTICE 'target_table: %', target_table;
+  --RAISE NOTICE 'target_table: %', target_table;
 
   -- return first boundary in intersections
   RETURN QUERY
@@ -418,12 +418,12 @@ BEGIN
   -- if no tables are found, raise notice and return null
   IF target_table IS NULL
   THEN
-    RAISE NOTICE 'No boundaries found for bounding box ''%'' in ''%''', ST_AsText(geom), boundary_id;
+    --RAISE NOTICE 'No boundaries found for bounding box ''%'' in ''%''', ST_AsText(geom), boundary_id;
     RETURN QUERY SELECT NULL::geometry, NULL::text;
     RETURN;
   END IF;
 
-  RAISE NOTICE 'target_table: %', target_table;
+  --RAISE NOTICE 'target_table: %', target_table;
 
   -- return first boundary in intersections
   RETURN QUERY

src/pg/sql/50_table_level_functions.sql

@@ -0,0 +1,119 @@
+CREATE TYPE cdb_observatory.ds_fdw_metadata as (schemaname text, tabname text, servername text);
+CREATE TYPE cdb_observatory.ds_return_metadata as (colnames text[], coltypes text[]);
+
+CREATE OR REPLACE FUNCTION cdb_observatory._OBS_ConnectUserTable(username text, orgname text, user_db_role text, input_schema text, dbname text, host_addr text, table_name text)
+RETURNS cdb_observatory.ds_fdw_metadata
+AS $$
+DECLARE
+  fdw_server text;
+  fdw_import_schema text;
+  connection_str json;
+  import_foreign_schema_q text;
+  epoch_timestamp text;
+BEGIN
+
+  SELECT extract(epoch from now() at time zone 'utc')::int INTO epoch_timestamp;
+  fdw_server := 'fdw_server_' || username || '_' || epoch_timestamp;
+  fdw_import_schema:= fdw_server;
+
+  -- Import foreign table
+  EXECUTE FORMAT ('SELECT cdb_observatory._OBS_ConnectRemoteTable(%L, %L, %L, %L, %L, %L, %L)', fdw_server, fdw_import_schema, dbname, host_addr, user_db_role, table_name, input_schema);
+
+  RETURN (fdw_import_schema::text, table_name::text, fdw_server::text);
+
+EXCEPTION
+  WHEN others THEN
+    -- Disconnect user imported table. Delete schema and FDW server.
+    EXECUTE 'DROP FOREIGN TABLE IF EXISTS ' || fdw_import_schema || '.' || table_name;
+    EXECUTE 'DROP SCHEMA IF EXISTS ' || fdw_import_schema || ' CASCADE';
+    EXECUTE 'DROP SERVER IF EXISTS ' || fdw_server || ' CASCADE;';
+    RETURN (null, null, null);
+END;
+$$ LANGUAGE plpgsql SECURITY DEFINER;
+
+CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetReturnMetadata(username text, orgname text, function_name text, params json)
+RETURNS cdb_observatory.ds_return_metadata
+AS $$
+DECLARE
+  colnames text[];
+  coltypes text[];
+  requested_measures text[];
+  measure text;
+BEGIN
+
+  -- Simple mock, there should be real logic in here.
+
+  IF $3 NOT ILIKE 'GetMeasure' OR $3 IS NULL THEN
+    RAISE 'This function is not supported yet: %', $3;
+  END IF;
+
+  SELECT translate($4::json->>'tag_name','[]', '{}')::text[] INTO requested_measures;
+
+  FOREACH measure IN ARRAY requested_measures
+  LOOP
+    IF NOT measure ILIKE ANY (Array['total_pop', 'pop_16_over']::text[]) THEN
+      RAISE 'This measure is not supported yet: %', measure;
+    END IF;
+  SELECT array_append(colnames, measure) INTO colnames;
+  SELECT array_append(coltypes, 'double precision'::text) INTO coltypes;
+
+  END LOOP;
+
+  RETURN (colnames::text[], coltypes::text[]);
+END;
+$$ LANGUAGE plpgsql;
+
+CREATE OR REPLACE FUNCTION cdb_observatory._OBS_FetchJoinFdwTableData(username text, orgname text, table_schema text, table_name text, function_name text, params json)
+RETURNS SETOF record
+AS $$
+DECLARE
+  data_query text;
+  tag_name text[];
+  tag text;
+  tags_list text;
+  tags_query text;
+  rec RECORD;
+BEGIN
+    SELECT translate($6::json->>'tag_name','[]', '{}')::text[] INTO tag_name;
+    SELECT array_to_string(tag_name, ',') INTO tags_list;
+    tags_query := '';
+
+    FOREACH tag IN ARRAY tag_name
+    LOOP
+      SELECT tags_query || ' sum(' || tag || '/fraction)::double precision as ' || tag || ', ' INTO tags_query;
+
+    END LOOP;
+
+    -- Simple mock, there should be real logic in here.
+    data_query := '(WITH _areas AS(SELECT ST_Area(a.the_geom::geography)'
+        || '/ (1000 * 1000) as fraction, a.geoid, b.cartodb_id FROM '
+        || 'observatory.obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308 as a, '
+        || table_schema || '.' || table_name || ' AS b '
+        || 'WHERE b.the_geom && a.the_geom ), values AS (SELECT geoid, '
+        || tags_list
+        || ' FROM observatory.obs_1a098da56badf5f32e336002b0a81708c40d29cd ) '
+        || 'SELECT '
+        || tags_query
+        || ' cartodb_id::int FROM _areas, values '
+        || 'WHERE values.geoid = _areas.geoid GROUP BY cartodb_id);';
+
+
+    FOR rec IN EXECUTE data_query
+    LOOP
+        RETURN NEXT rec;
+    END LOOP;
+    RETURN;
+END;
+$$ LANGUAGE plpgsql SECURITY DEFINER;
+
+
+CREATE OR REPLACE FUNCTION cdb_observatory._OBS_DisconnectUserTable(username text, orgname text, table_schema text, table_name text, servername text)
+RETURNS boolean
+AS $$
+BEGIN
+    EXECUTE 'DROP FOREIGN TABLE IF EXISTS "' || table_schema || '".' || table_name;
+    EXECUTE 'DROP SCHEMA IF EXISTS ' || table_schema || ' CASCADE';
+    EXECUTE 'DROP SERVER IF EXISTS ' || servername || ' CASCADE;';
+    RETURN true;
+END;
+$$ LANGUAGE plpgsql SECURITY DEFINER;

src/pg/test/expected/40_observatory_utility_test.out

@@ -9,21 +9,12 @@ t
 _obs_geomtable_with_null_response
 t
 (1 row)
-test_get_obs_column_with_geoid_and_census_1|test_get_obs_column_with_geoid_and_census_2
-t|t
-(1 row)
-obs_getcolumndata_missing_measure
-t
-(1 row)
 _obs_buildsnapshotquery_test_1
 t
 (1 row)
 _obs_buildsnapshotquery_test_2
 t
 (1 row)
-_obs_getrelatedcolumn_test
-t
-(1 row)
 _obs_standardizemeasurename_test
 t
 (1 row)

src/pg/test/expected/41_observatory_augmentation_test.out

@@ -1,5 +1,4 @@
-\i test/fixtures/load_fixtures.sql
-SET client_min_messages TO WARNING;
+\pset format unaligned
 \set ECHO none
 obs_getdemographicsnapshot_test_no_returns
 t

src/pg/test/expected/42_observatory_exploration_test.out

@@ -1,5 +1,4 @@
-\i test/fixtures/load_fixtures.sql
-SET client_min_messages TO WARNING;
+\pset format unaligned
 \set ECHO none
 _obs_searchtables_tables_match|_obs_searchtables_timespan_matches
 t|t

src/pg/test/expected/44_observatory_geometries_test.out

@@ -1,7 +1,4 @@
 \pset format unaligned
-\set ECHO all
-\i test/fixtures/load_fixtures.sql
-SET client_min_messages TO WARNING;
 \set ECHO none
 obs_getboundary_cartodb_census_tract
 t

src/pg/test/fixtures/drop_fixtures.sql

@@ -20,3 +20,5 @@ DROP TABLE IF EXISTS observatory.obs_6c1309a64d8f3e6986061f4d1ca7b57743e75e74;
 DROP TABLE IF EXISTS observatory.obs_d39f7fe5959891c8296490d83c22ded31c54af13;
 DROP TABLE IF EXISTS observatory.obs_144e8b4f906885b2e057ac4842644a553ae49c6e;
 DROP TABLE IF EXISTS observatory.obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308;
+
+DROP TABLE IF EXISTS observatory.obs_meta;

src/pg/test/sql/40_observatory_utility_test.sql

@@ -1,6 +1,8 @@
 \pset format unaligned
 \set ECHO all
 \i test/fixtures/load_fixtures.sql
+SET client_min_messages TO WARNING;
+\set ECHO none
 
 -- OBS_GeomTable
 -- get table with known geometry_id
@@ -27,29 +29,6 @@ SELECT
 --     'us.census.tiger.census_tract'
 --   );
 
-WITH result as (
-SELECT
-  array_agg(a) expected from cdb_observatory._OBS_GetColumnData(
-    'us.census.tiger.census_tract',
-    Array['us.census.spielman_singleton_segments.X55', 'us.census.acs.B01003001'],
-         '2010 - 2014') a
-)
-select
-(expected)[1]::text  = '{"colname":"x55","tablename":"obs_65f29658e096ca1485bf683f65fdbc9f05ec3c5d","aggregate":null,"name":"Spielman-Singleton Segments: 55 Clusters","type":"Text","description":"Sociodemographic classes from Spielman and Singleton 2015, 55 clusters","boundary_id":"us.census.tiger.census_tract"}' as test_get_obs_column_with_geoid_and_census_1,
-(expected)[2]::text  = '{"colname":"total_pop","tablename":"obs_b393b5b88c6adda634b2071a8005b03c551b609a","aggregate":"sum","name":"Total Population","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates.","boundary_id":"us.census.tiger.census_tract"}' as test_get_obs_column_with_geoid_and_census_2
-from result;
-
--- should be null-valued
-WITH result as (
-SELECT
-  array_agg(a) expected from cdb_observatory._OBS_GetColumnData(
-    'us.census.tiger.census_tract',
-    Array['us.census.tiger.baloney'],
-    '2010 - 2014') a
-)
-select expected is null as OBS_GetColumnData_missing_measure
-from result;
-
 -- OBS_BuildSnapshotQuery
 -- Should give back: SELECT  vals[1] As total_pop, vals[2] As male_pop, vals[3] As female_pop, vals[4] As median_age
 SELECT
@@ -63,19 +42,8 @@ SELECT
     Array['mandarin_orange']
   ) = 'SELECT  vals[1] As mandarin_orange' As _OBS_BuildSnapshotQuery_test_2;
 
-SELECT cdb_observatory._OBS_GetRelatedColumn(
-    Array[
-     'es.ine.t3_1',
-     'us.census.acs.B01003001',
-     'us.census.acs.B01001002'
-    ],
-     'denominator'
- ) = '{es.ine.t1_1,NULL,us.census.acs.B01003001}' As _OBS_GetRelatedColumn_test;
-
 -- should give back a standardized measure name
 SELECT cdb_observatory._OBS_StandardizeMeasureName('test 343 %% 2 qqq }}{{}}') = 'test_343_2_qqq' As _OBS_StandardizeMeasureName_test;
 
 SELECT cdb_observatory.OBS_DumpVersion()
   IS NOT NULL AS OBS_DumpVersion_notnull;
-
-\i test/fixtures/drop_fixtures.sql

src/pg/test/sql/41_observatory_augmentation_test.sql

@@ -1,6 +1,6 @@
-\i test/fixtures/load_fixtures.sql
 \pset format unaligned
 \set ECHO none
+SET client_min_messages TO WARNING;
 
 --
 WITH result as(
@@ -131,7 +131,7 @@ WITH result as (
 -- Point-based OBS_GetMeasure with zillow
 SELECT abs(OBS_GetMeasure_zhvi_point - 583600) / 583600 < 0.001 AS OBS_GetMeasure_zhvi_point_test FROM cdb_observatory.OBS_GetMeasure(
   ST_SetSRID(ST_Point(-73.94602417945862, 40.6768220087458), 4326),
-  'us.zillow.AllHomes_Zhvi', 'area', 'us.census.tiger.zcta5', '2014-01'
+  'us.zillow.AllHomes_Zhvi', null, 'us.census.tiger.zcta5', '2014-01'
 ) As t(OBS_GetMeasure_zhvi_point);
 
 -- Point-based OBS_GetMeasure with zillow default to latest
@@ -172,7 +172,7 @@ SELECT cdb_observatory.OBS_GetCategory(
 
 -- Poly-based OBS_GetCategory
 SELECT cdb_observatory.OBS_GetCategory(
-  cdb_observatory._TestArea(), 'us.census.spielman_singleton_segments.X10') = 'Low income, mix of minorities' As obs_getcategory_polygon;
+  cdb_observatory._TestArea(), 'us.census.spielman_singleton_segments.X10') = 'Wealthy, urban without Kids' As obs_getcategory_polygon;
 
 -- Point-based OBS_GetPopulation, default normalization (area)
 SELECT (abs(OBS_GetPopulation - 10923.093200390833950) / 10923.093200390833950) < 0.001 As OBS_GetPopulation FROM
@@ -201,7 +201,7 @@ SELECT cdb_observatory.OBS_GetUSCensusCategory(
 
 -- Area-based OBS_GetUSCensusCategory
 SELECT cdb_observatory.OBS_GetUSCensusCategory(
-  cdb_observatory._testarea(), 'Spielman-Singleton Segments: 10 Clusters') = 'Low income, mix of minorities' As OBS_GetUSCensusCategory_polygon;
+  cdb_observatory._testarea(), 'Spielman-Singleton Segments: 10 Clusters') = 'Wealthy, urban without Kids' As OBS_GetUSCensusCategory_polygon;
 
 
 -- OBS_GetMeasureById tests
@@ -236,5 +236,3 @@ SELECT cdb_observatory.OBS_GetMeasureById(
   'us.census.tiger.block_group',
   '2010 - 2014'
 ) IS NULL As OBS_GetMeasureById_nulls;
-
-\i test/fixtures/drop_fixtures.sql

src/pg/test/sql/42_observatory_exploration_test.sql

@@ -1,5 +1,6 @@
-\i test/fixtures/load_fixtures.sql
 \pset format unaligned
+\set ECHO none
+SET client_min_messages TO WARNING;
 
 -- set up variables for use in testing
 
@@ -32,5 +33,3 @@ SELECT COUNT(*) > 0 AS _OBS_GetAvailableBoundariesExist
 FROM cdb_observatory.OBS_GetAvailableBoundaries(
   cdb_observatory._TestPoint()
 ) AS t(boundary_id, description, time_span, tablename);
-
-\i test/fixtures/drop_fixtures.sql

src/pg/test/sql/44_observatory_geometries_test.sql

@@ -1,6 +1,6 @@
 \pset format unaligned
-\set ECHO all
-\i test/fixtures/load_fixtures.sql
+\set ECHO none
+SET client_min_messages TO WARNING;
 
 -- set up variables for use in testing