cartodb4/observatory-extension
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #147 from CartoDB/develop

Browse Source 
Release 1.0.2
This commit is contained in:
Mario de Frutos
2016-07-15 15:28:52 +02:00
committed by GitHub
parent 8fa2d642bf 588cda3262
commit 25570e5b11
20 changed files with 12541 additions and 8708 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
NEWS.md
View File

@@ -1,3 +1,30 @@
1.0.2 (2016-07-12)
---
__Bugfixes__
* Fix for `OBS_GetCategory` outside the US ([#135](https://github.com/CartoDB/observatory-extension/pull/137))
* `OBS_GetMeasure` now respects the `normalize` parameter even when passed
a multi/polygon. Previously, no normalization was erroneously assumed.
__Improvements__
* Automated tests cover Mexico data
* `obs_meta` is now provisioned during unit tests
* `obs_meta` is now used during end-to-end tests
* `OBS_GetMeasureByID` uses `obs_meta` internally, which should help
performance
* `OBS_GetCategory` uses `obs_meta` internally, which should help perfromance
* `OBS_GetCategory` will pick the correct category for an arbitrary polygon
(the category covering the highest % of that polygon)
* `OBS_GetMeasure` has been updated to use `obs_meta` internally, which should
help performance
* `OBS_GetMeasure` now can be passed "none" and skip normalization by area or
denominator for points
* Fixtures are only loaded at the start of the unit test suite, and dropped at the end,
instead of at the start/end of each individual test file
* Comment noisy NOTICEs ([#73](https://github.com/CartoDB/observatory-extension/issues/73))
1.0.1 (2016-07-01)
---

4
doc/boundary_functions.md
View File

@@ -1,8 +1,8 @@
# Boundary Functions
Use the following functions to retrieve [Boundary](/cartodb-platform/data/overview/#boundary-data) data. Data ranges from small areas (e.g. US Census Block Groups) to large areas (e.g. Countries). You can access boundaries by point location lookup, bounding box lookup, direct ID access and several other methods described below.
Use the following functions to retrieve [Boundary](/carto-engine/data/overview/#boundary-data) data. Data ranges from small areas (e.g. US Census Block Groups) to large areas (e.g. Countries). You can access boundaries by point location lookup, bounding box lookup, direct ID access and several other methods described below.
You can [access](/cartodb-platform/data/accessing/#accessing-the-data-observatory) boundaries through the CartoDB Editor. The same methods will work if you are using the CartoDB Platform to develop your application. We [encourage you](/cartodb-platform/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
You can [access](/carto-engine/data/accessing/#accessing-the-data-observatory) boundaries through the CARTO Editor. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
## OBS_GetBoundariesByGeometry(polygon geometry, geometry_id text)

4
doc/discovery_functions.md
View File

@@ -1,6 +1,6 @@
# Discovery Functions
If you are using the [discovery methods](/cartodb-platform/data/overview/#discovery-methods) from the Data Observatory, use the following functions to retrieve [boundary](/cartodb-platform/data/overview/#boundary-data) and [measures](/cartodb-platform/data/overview/#measures-data) data.
If you are using the [discovery methods](/carto-engine/data/overview/#discovery-methods) from the Data Observatory, use the following functions to retrieve [boundary](/carto-engine/data/overview/#boundary-data) and [measures](/carto-engine/data/overview/#measures-data) data.
## OBS_Search(search_term)
@@ -47,7 +47,7 @@ A TABLE containing the following properties
Key | Description
--- | ---
boundary_id | a boundary identifier from the [boundary ID glossary](/cartodb-platform/data/glossary/#boundary-ids)
boundary_id | a boundary identifier from the [boundary ID glossary](/carto-engine/data/glossary/#boundary-ids)
description | a brief description of the boundary dataset
time_span | the timespan attached the boundary. this does not mean that the boundary is invalid outside of the timespan, but is the explicit timespan published with the geometry.

4
doc/measures_functions.md
View File

@@ -1,10 +1,10 @@
# Measures Functions
[Data Observatory Measures](/cartodb-platform/data/overview/#measures-methods) are the numerical location data you can access. The measure functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (point or polygon) and receiving back a measure (an absolute value) for that location.
[Data Observatory Measures](/carto-engine/data/overview/#measures-methods) are the numerical location data you can access. The measure functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (point or polygon) and receiving back a measure (an absolute value) for that location.
There are hundreds of measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf).
You can [access](/cartodb-platform/data/accessing/#accessing-the-data-observatory) measures through the CartoDB Editor. The same methods will work if you are using the CartoDB Platform to develop your application. We [encourage you](/cartodb-platform/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
You can [access](/carto-engine/data/accessing/#accessing-the-data-observatory) measures through the CARTO Editor. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
## OBS_GetUSCensusMeasure(point geometry, measure_name text)

37
scripts/autotest.py
View File

@@ -24,17 +24,18 @@ def query(q, is_meta=False, **options):
params['api_key'] = META_API_KEY if is_meta else API_KEY
return requests.get(url, params=params)
MEASURE_COLUMNS = [(r['id'], r['point_only'], ) for r in query('''
SELECT id, aggregate NOT ILIKE 'sum' as point_only
FROM obs_column
WHERE type ILIKE 'numeric'
AND weight > 0
MEASURE_COLUMNS = [(r['numer_id'], r['point_only'], ) for r in query('''
SELECT distinct numer_id, numer_aggregate NOT ILIKE 'sum' as point_only
FROM obs_meta
WHERE numer_type ILIKE 'numeric'
AND numer_weight > 0
''', is_meta=True).json()['rows']]
CATEGORY_COLUMNS = [(r['id'], ) for r in query('''
SELECT id FROM obs_column
WHERE type ILIKE 'text'
AND weight > 0
CATEGORY_COLUMNS = [(r['numer_id'], ) for r in query('''
SELECT distinct numer_id
FROM obs_meta
WHERE numer_type ILIKE 'text'
AND numer_weight > 0
''', is_meta=True).json()['rows']]
BOUNDARY_COLUMNS = [(r['id'], ) for r in query('''
@@ -43,12 +44,12 @@ WHERE type ILIKE 'geometry'
AND weight > 0
''', is_meta=True).json()['rows']]
US_CENSUS_MEASURE_COLUMNS = [(r['name'], ) for r in query('''
SELECT c.name FROM obs_column c, obs_column_tag ct
WHERE type ILIKE 'numeric'
AND c.id = ct.column_id
AND ct.tag_id LIKE 'us.census%'
AND weight > 0
US_CENSUS_MEASURE_COLUMNS = [(r['numer_name'], ) for r in query('''
SELECT distinct numer_name
FROM obs_meta
WHERE numer_type ILIKE 'numeric'
AND 'us.census.acs.acs' = ANY (subsection_tags)
AND numer_weight > 0
''', is_meta=True).json()['rows']]
@@ -95,6 +96,8 @@ def default_point(column_id):
return 'CDB_LatLng(42.8226119029222, -2.51141249535454)'
elif column_id.startswith('us.zillow'):
return 'CDB_LatLng(28.3305906291771, -81.3544048197256)'
elif column_id.startswith('mx.'):
return 'CDB_LatLng(19.41347699386547, -99.17019367218018)'
else:
return 'CDB_LatLng(40.7, -73.9)'
@@ -110,6 +113,7 @@ def default_area(column_id):
@parameterized(US_CENSUS_MEASURE_COLUMNS)
def test_get_us_census_measure_points(name):
print 'test_get_us_census_measure_points, ', name
resp = query('''
SELECT * FROM {schema}OBS_GetUSCensusMeasure({point}, '{name}')
'''.format(name=name.replace("'", "''"),
@@ -123,6 +127,7 @@ SELECT * FROM {schema}OBS_GetUSCensusMeasure({point}, '{name}')
@parameterized(MEASURE_COLUMNS)
def test_get_measure_areas(column_id, point_only):
print 'test_get_measure_areas, ', column_id, point_only
if point_only:
return
resp = query('''
@@ -138,6 +143,7 @@ SELECT * FROM {schema}OBS_GetMeasure({area}, '{column_id}')
@parameterized(MEASURE_COLUMNS)
def test_get_measure_points(column_id, point_only):
print 'test_get_measure_points, ', column_id, point_only
resp = query('''
SELECT * FROM {schema}OBS_GetMeasure({point}, '{column_id}')
'''.format(column_id=column_id,
@@ -162,6 +168,7 @@ SELECT * FROM {schema}OBS_GetMeasure({point}, '{column_id}')
@parameterized(CATEGORY_COLUMNS)
def test_get_category_points(column_id):
print 'test_get_category_points, ', column_id
resp = query('''
SELECT * FROM {schema}OBS_GetCategory({point}, '{column_id}')
'''.format(column_id=column_id,

123
scripts/generate_fixtures.py
View File

@@ -34,7 +34,7 @@ def select_star(tablename):
cdb = Dumpr('observatory.cartodb.com','')
metadata = ['obs_table', 'obs_column_table', 'obs_column', 'obs_column_tag',
'obs_tag', 'obs_column_to_column', 'obs_dump_version']
'obs_tag', 'obs_column_to_column', 'obs_dump_version', ]
fixtures = [
('us.census.tiger.census_tract', 'us.census.tiger.census_tract', '2014'),
@@ -90,3 +90,124 @@ with open('src/pg/test/fixtures/load_fixtures.sql', 'w') as outfile:
cdb.dump(' '.join([select_star(tablename), "WHERE {}::text {} {}".format(colname, compare, where)]),
tablename, outfile, schema='observatory')
dropfiles.write('DROP TABLE IF EXISTS observatory.{};\n'.format(tablename))
outfile.write('''
ALTER TABLE observatory.obs_table
ADD PRIMARY KEY (id);
ALTER TABLE observatory.obs_column_table
ADD PRIMARY KEY (column_id, table_id);
CREATE UNIQUE INDEX ON observatory.obs_column_table (table_id, column_id);
CREATE UNIQUE INDEX ON observatory.obs_column_table (table_id, colname);
ALTER TABLE observatory.obs_column
ADD PRIMARY KEY (id);
ALTER TABLE observatory.obs_column_to_column
ADD PRIMARY KEY (source_id, target_id, reltype);
CREATE UNIQUE INDEX ON observatory.obs_column_to_column (target_id, source_id, reltype);
CREATE INDEX ON observatory.obs_column_to_column (reltype);
ALTER TABLE observatory.obs_column_tag
ADD PRIMARY KEY (column_id, tag_id);
CREATE UNIQUE INDEX ON observatory.obs_column_tag (tag_id, column_id);
ALTER TABLE observatory.obs_tag
ADD PRIMARY KEY (id);
CREATE INDEX ON observatory.obs_tag (type);
VACUUM ANALYZE observatory.obs_table;
VACUUM ANALYZE observatory.obs_column_table;
VACUUM ANALYZE observatory.obs_column;
VACUUM ANALYZE observatory.obs_column_to_column;
VACUUM ANALYZE observatory.obs_column_tag;
VACUUM ANALYZE observatory.obs_tag;
CREATE TABLE observatory.obs_meta AS
SELECT numer_c.id numer_id,
denom_c.id denom_id,
geom_c.id geom_id,
MAX(numer_c.name) numer_name,
MAX(denom_c.name) denom_name,
MAX(geom_c.name) geom_name,
MAX(numer_c.description) numer_description,
MAX(denom_c.description) denom_description,
MAX(geom_c.description) geom_description,
MAX(numer_c.aggregate) numer_aggregate,
MAX(denom_c.aggregate) denom_aggregate,
MAX(geom_c.aggregate) geom_aggregate,
MAX(numer_c.type) numer_type,
MAX(denom_c.type) denom_type,
MAX(geom_c.type) geom_type,
MAX(numer_data_ct.colname) numer_colname,
MAX(denom_data_ct.colname) denom_colname,
MAX(geom_geom_ct.colname) geom_colname,
MAX(numer_geomref_ct.colname) numer_geomref_colname,
MAX(denom_geomref_ct.colname) denom_geomref_colname,
MAX(geom_geomref_ct.colname) geom_geomref_colname,
MAX(numer_t.tablename) numer_tablename,
MAX(denom_t.tablename) denom_tablename,
MAX(geom_t.tablename) geom_tablename,
MAX(numer_t.timespan) numer_timespan,
MAX(denom_t.timespan) denom_timespan,
MAX(numer_c.weight) numer_weight,
MAX(denom_c.weight) denom_weight,
MAX(geom_c.weight) geom_weight,
MAX(geom_t.timespan) geom_timespan,
MAX(geom_t.the_geom_webmercator)::geometry AS the_geom_webmercator,
ARRAY_AGG(DISTINCT s_tag.id) section_tags,
ARRAY_AGG(DISTINCT ss_tag.id) subsection_tags,
ARRAY_AGG(DISTINCT unit_tag.id) unit_tags
FROM observatory.obs_column_table numer_data_ct,
observatory.obs_table numer_t,
observatory.obs_column_table numer_geomref_ct,
observatory.obs_column geomref_c,
observatory.obs_column_to_column geomref_c2c,
observatory.obs_column geom_c,
observatory.obs_column_table geom_geom_ct,
observatory.obs_column_table geom_geomref_ct,
observatory.obs_table geom_t,
observatory.obs_column_tag ss_ctag,
observatory.obs_tag ss_tag,
observatory.obs_column_tag s_ctag,
observatory.obs_tag s_tag,
observatory.obs_column_tag unit_ctag,
observatory.obs_tag unit_tag,
observatory.obs_column numer_c
LEFT JOIN (
observatory.obs_column_to_column denom_c2c
JOIN observatory.obs_column denom_c ON denom_c2c.target_id = denom_c.id
JOIN observatory.obs_column_table denom_data_ct ON denom_data_ct.column_id = denom_c.id
JOIN observatory.obs_table denom_t ON denom_data_ct.table_id = denom_t.id
JOIN observatory.obs_column_table denom_geomref_ct ON denom_geomref_ct.table_id = denom_t.id
) ON denom_c2c.source_id = numer_c.id
WHERE numer_c.id = numer_data_ct.column_id
AND numer_data_ct.table_id = numer_t.id
AND numer_t.id = numer_geomref_ct.table_id
AND numer_geomref_ct.column_id = geomref_c.id
AND geomref_c2c.reltype = 'geom_ref'
AND geomref_c.id = geomref_c2c.source_id
AND geom_c.id = geomref_c2c.target_id
AND geom_geomref_ct.column_id = geomref_c.id
AND geom_geomref_ct.table_id = geom_t.id
AND geom_geom_ct.column_id = geom_c.id
AND geom_geom_ct.table_id = geom_t.id
AND geom_c.type ILIKE 'geometry'
AND numer_c.type NOT ILIKE 'geometry'
AND numer_t.id != geom_t.id
AND numer_c.id != geomref_c.id
AND unit_tag.type = 'unit'
AND ss_tag.type = 'subsection'
AND s_tag.type = 'section'
AND unit_ctag.column_id = numer_c.id
AND unit_ctag.tag_id = unit_tag.id
AND ss_ctag.column_id = numer_c.id
AND ss_ctag.tag_id = ss_tag.id
AND s_ctag.column_id = numer_c.id
AND s_ctag.tag_id = s_tag.id
AND (denom_c2c.reltype = 'denominator' OR denom_c2c.reltype IS NULL)
AND (denom_geomref_ct.column_id = geomref_c.id OR denom_geomref_ct.column_id IS NULL)
AND (denom_t.timespan = numer_t.timespan OR denom_t.timespan IS NULL)
GROUP BY numer_c.id, denom_c.id, geom_c.id,
numer_t.id, denom_t.id, geom_t.id;
''')
dropfiles.write('''
DROP TABLE IF EXISTS observatory.obs_meta;
''')

2
src/pg/observatory.control
View File

@@ -1,5 +1,5 @@
comment = 'CartoDB Observatory backend extension'
default_version = '1.0.1'
default_version = '1.0.2'
requires = 'postgis'
superuser = true
schema = cdb_observatory

22
src/pg/sql/40_observatory_utility.sql
View File

@@ -158,28 +158,6 @@ BEGIN
END;
$$ LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetRelatedColumn(columns_ids text[], reltype text )
RETURNS TEXT[]
AS $$
DECLARE
result TEXT[];
BEGIN
EXECUTE '
With ids as (
select row_number() over() as no, id from (select unnest($1) as id) t
)
select array_agg(target_id order by no)
FROM ids
LEFT JOIN observatory.obs_column_to_column
on source_id = id
where reltype = $2 or reltype is null
'
INTO result
using columns_ids, reltype;
return result;
END;
$$ LANGUAGE plpgsql;
-- Function that replaces all non digits or letters with _ trims and lowercases the
-- passed measure name

298
src/pg/sql/41_observatory_augmentation.sql
View File

@@ -169,19 +169,19 @@ BEGIN
IF geom_table_name IS NULL
THEN
RAISE NOTICE 'Point % is outside of the data region', ST_AsText(geom);
--raise notice 'Point % is outside of the data region', ST_AsText(geom);
-- TODO this should return JSON
RETURN QUERY SELECT '{}'::json;
RETURN;
END IF;
IF data_table_info IS NULL THEN
RAISE NOTICE 'Cannot find data table for boundary ID %, column_ids %, and time_span %', geometry_level, column_ids, time_span;
--raise notice 'Cannot find data table for boundary ID %, column_ids %, and time_span %', geometry_level, column_ids, time_span;
END IF;
IF ST_GeometryType(geom) = 'ST_Point'
THEN
RAISE NOTICE 'geom_table_name %, data_table_info %', geom_table_name, data_table_info::json[];
--raise notice 'geom_table_name %, data_table_info %', geom_table_name, data_table_info::json[];
results := cdb_observatory._OBS_GetPoints(geom,
geom_table_name,
data_table_info);
@@ -260,7 +260,7 @@ BEGIN
USING geom
INTO geoid;
RAISE NOTICE 'geoid is %, geometry table is % ', geoid, geom_table_name;
--raise notice 'geoid is %, geometry table is % ', geoid, geom_table_name;
EXECUTE
format('SELECT ST_Area(the_geom::geography) / (1000 * 1000)
@@ -273,7 +273,7 @@ BEGIN
IF area IS NULL
THEN
RAISE NOTICE 'No geometry at %', ST_AsText(geom);
--raise notice 'No geometry at %', ST_AsText(geom);
END IF;
query := 'SELECT Array[';
@@ -338,44 +338,173 @@ $$ LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetMeasure(
geom geometry(Geometry, 4326),
measure_id TEXT,
normalize TEXT DEFAULT 'area', -- TODO none/null
normalize TEXT DEFAULT NULL,
boundary_id TEXT DEFAULT NULL,
time_span TEXT DEFAULT NULL
)
RETURNS NUMERIC
AS $$
DECLARE
geom_type TEXT;
map_type TEXT;
numer_aggregate TEXT;
numer_colname TEXT;
numer_geomref_colname TEXT;
numer_tablename TEXT;
denom_colname TEXT;
denom_geomref_colname TEXT;
denom_tablename TEXT;
geom_colname TEXT;
geom_geomref_colname TEXT;
geom_tablename TEXT;
result NUMERIC;
measure_ids TEXT[];
denominator_id TEXT;
vals NUMERIC[];
sql TEXT;
numer_name TEXT;
BEGIN
IF normalize ILIKE 'area' THEN
measure_ids := ARRAY[measure_id];
EXECUTE
$query$
SELECT numer_aggregate, numer_colname, numer_geomref_colname, numer_tablename,
denom_colname, denom_geomref_colname, denom_tablename,
geom_colname, geom_geomref_colname, geom_tablename, numer_name
FROM observatory.obs_meta
WHERE (geom_id = $1 OR ($1 = ''))
AND numer_id = $2
AND (numer_timespan = $3 OR ($3 = ''))
ORDER BY geom_weight DESC, numer_timespan DESC
LIMIT 1
$query$
INTO numer_aggregate, numer_colname, numer_geomref_colname, numer_tablename,
denom_colname, denom_geomref_colname, denom_tablename,
geom_colname, geom_geomref_colname, geom_tablename, numer_name
USING COALESCE(boundary_id, ''), measure_id, COALESCE(time_span, '');
IF ST_GeometryType(geom) = 'ST_Point' THEN
geom_type := 'point';
ELSIF ST_GeometryType(geom) IN ('ST_Polygon', 'ST_MultiPolygon') THEN
geom_type := 'polygon';
ELSE
RAISE EXCEPTION 'Invalid geometry type (%), can only handle ''ST_Point'', ''ST_Polygon'', and ''ST_MultiPolygon''',
ST_GeometryType(geom);
END IF;
IF normalize ILIKE 'area' AND numer_aggregate ILIKE 'sum' THEN
map_type := 'areaNormalized';
ELSIF normalize ILIKE 'denominator' THEN
EXECUTE 'SELECT (cdb_observatory._OBS_GetRelatedColumn(ARRAY[$1], ''denominator''))[1]
' INTO denominator_id
USING measure_id;
measure_ids := ARRAY[measure_id, denominator_id];
ELSIF normalize ILIKE 'none' THEN
-- TODO we need a switch on obs_get to disable area normalization
RAISE EXCEPTION 'No normalization not yet supported.';
map_type := 'denominated';
ELSE
RAISE EXCEPTION 'Only valid inputs for "normalize" are "area" (default) and "denominator".';
-- defaults: area normalization for point if it's possible and none for
-- polygon or non-summable point
IF geom_type = 'point' AND numer_aggregate ILIKE 'sum' THEN
map_type := 'areaNormalized';
ELSE
map_type := 'predenominated';
END IF;
END IF;
EXECUTE '
SELECT ARRAY_AGG(val) FROM (SELECT (cdb_observatory._OBS_Get($1, $2, $3, $4)->>''value'')::NUMERIC val) b
'
INTO vals
USING geom, measure_ids, time_span, boundary_id;
IF normalize ILIKE 'denominator' THEN
RETURN (vals)[1]/(vals)[2];
ELSE
RETURN (vals)[1];
IF geom_type = 'point' THEN
IF map_type = 'areaNormalized' THEN
sql = format('WITH _geom AS (SELECT ST_Area(geom.%I::Geography) / 1000000 area, geom.%I geom_ref
FROM observatory.%I geom
WHERE ST_Within(%L, geom.%I)
LIMIT 1)
SELECT numer.%I / (SELECT area FROM _geom)
FROM observatory.%I numer
WHERE numer.%I = (SELECT geom_ref FROM _geom)',
geom_colname, geom_geomref_colname, geom_tablename,
geom, geom_colname, numer_colname, numer_tablename,
numer_geomref_colname);
ELSIF map_type = 'denominated' THEN
sql = format('SELECT numer.%I / NULLIF((SELECT denom.%I FROM observatory.%I denom WHERE denom.%I = numer.%I LIMIT 1), 0)
FROM observatory.%I numer
WHERE numer.%I = (SELECT geom.%I FROM observatory.%I geom WHERE ST_Within(%L, geom.%I) LIMIT 1)',
numer_colname, denom_colname, denom_tablename,
denom_geomref_colname, numer_geomref_colname,
numer_tablename,
numer_geomref_colname, geom_geomref_colname,
geom_tablename, geom, geom_colname);
ELSIF map_type = 'predenominated' THEN
sql = format('SELECT numer.%I
FROM observatory.%I numer
WHERE numer.%I = (SELECT geom.%I FROM observatory.%I geom WHERE ST_Within(%L, geom.%I) LIMIT 1)',
numer_colname, numer_tablename,
numer_geomref_colname, geom_geomref_colname, geom_tablename,
geom, geom_colname);
END IF;
ELSIF geom_type = 'polygon' THEN
IF map_type = 'areaNormalized' THEN
sql = format('WITH _geom AS (SELECT ST_Area(ST_Intersection(%L, geom.%I))
/ ST_Area(geom.%I) overlap, geom.%I geom_ref
FROM observatory.%I geom
WHERE ST_Intersects(%L, geom.%I)
AND ST_Area(ST_Intersection(%L, geom.%I)) / ST_Area(geom.%I) > 0)
SELECT SUM(numer.%I * (SELECT _geom.overlap FROM _geom WHERE _geom.geom_ref = numer.%I)) /
ST_Area(%L::Geography)
FROM observatory.%I numer
WHERE numer.%I = ANY ((SELECT ARRAY_AGG(geom_ref) FROM _geom)::TEXT[])',
geom, geom_colname, geom_colname,
geom_geomref_colname, geom_tablename,
geom, geom_colname,
geom, geom_colname, geom_colname,
numer_colname, numer_geomref_colname,
geom, numer_tablename,
numer_geomref_colname);
ELSIF map_type = 'denominated' THEN
sql = format('WITH _geom AS (SELECT ST_Area(ST_Intersection(%L, geom.%I))
/ ST_Area(geom.%I) overlap, geom.%I geom_ref
FROM observatory.%I geom
WHERE ST_Intersects(%L, geom.%I)
AND ST_Area(ST_Intersection(%L, geom.%I)) / ST_Area(geom.%I) > 0),
_denom AS (SELECT denom.%I, denom.%I geom_ref
FROM observatory.%I denom
WHERE denom.%I = ANY ((SELECT ARRAY_AGG(geom_ref) FROM _geom)::TEXT[]))
SELECT SUM(numer.%I * (SELECT _geom.overlap FROM _geom WHERE _geom.geom_ref = numer.%I)) /
SUM((SELECT _denom.%I * (SELECT _geom.overlap
FROM _geom
WHERE _geom.geom_ref = _denom.geom_ref)
FROM _denom WHERE _denom.geom_ref = numer.%I))
FROM observatory.%I numer
WHERE numer.%I = ANY ((SELECT ARRAY_AGG(geom_ref) FROM _geom)::TEXT[])',
geom, geom_colname,
geom_colname, geom_geomref_colname,
geom_tablename,
geom, geom_colname,
geom, geom_colname, geom_colname,
denom_colname, denom_geomref_colname,
denom_tablename,
denom_geomref_colname,
numer_colname, numer_geomref_colname,
denom_colname,
numer_geomref_colname,
numer_tablename,
numer_geomref_colname);
ELSIF map_type = 'predenominated' THEN
IF numer_aggregate NOT ILIKE 'sum' THEN
RAISE EXCEPTION 'Cannot calculate "%" (%) for custom area as it cannot be summed, use ST_PointOnSurface instead',
numer_name, numer_id;
ELSE
sql = format('WITH _geom AS (SELECT ST_Area(ST_Intersection(%L, geom.%I))
/ ST_Area(geom.%I) overlap, geom.%I geom_ref
FROM observatory.%I geom
WHERE ST_Intersects(%L, geom.%I)
AND ST_Area(ST_Intersection(%L, geom.%I)) / ST_Area(geom.%I) > 0)
SELECT SUM(numer.%I * (SELECT _geom.overlap FROM _geom WHERE _geom.geom_ref = numer.%I))
FROM observatory.%I numer
WHERE numer.%I = ANY ((SELECT ARRAY_AGG(geom_ref) FROM _geom)::TEXT[])',
geom, geom_colname, geom_colname,
geom_geomref_colname, geom_tablename,
geom, geom_colname,
geom, geom_colname, geom_colname,
numer_colname, numer_geomref_colname,
numer_tablename,
numer_geomref_colname);
END IF;
END IF;
END IF;
EXECUTE sql INTO result;
RETURN result;
END;
$$ LANGUAGE plpgsql;
@@ -393,33 +522,30 @@ DECLARE
colname TEXT;
measure_val NUMERIC;
data_geoid_colname TEXT;
test_query TEXT;
BEGIN
SELECT x ->> 'colname', x ->> 'tablename' INTO colname, target_table
FROM cdb_observatory._OBS_GetColumnData(boundary_id, Array[measure_id], time_span) As x;
EXECUTE
format('SELECT ct.colname
FROM observatory.obs_column_to_column c2c,
observatory.obs_column_table ct,
observatory.obs_table t
WHERE c2c.reltype = ''geom_ref''
AND ct.column_id = c2c.source_id
AND ct.table_id = t.id
AND t.tablename = %L'
, target_table)
INTO data_geoid_colname;
$query$
SELECT numer_colname, numer_geomref_colname, numer_tablename
FROM observatory.obs_meta
WHERE (geom_id = $1 OR ($1 = ''))
AND numer_id = $2
AND (numer_timespan = $3 OR ($3 = ''))
ORDER BY geom_weight DESC, numer_timespan DESC
LIMIT 1
$query$
INTO colname, data_geoid_colname, target_table
USING COALESCE(boundary_id, ''), measure_id, COALESCE(time_span, '');
RAISE DEBUG 'target_table %, colname %', target_table, colname;
--RAISE DEBUG 'target_table %, colname %', target_table, colname;
EXECUTE format(
'SELECT %I
FROM observatory.%I
WHERE %I.%I = %L',
FROM observatory.%I data
WHERE data.%I = %L',
colname,
target_table,
target_table, data_geoid_colname, geom_ref)
data_geoid_colname, geom_ref)
INTO measure_val;
RETURN measure_val;
@@ -436,27 +562,61 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetCategory(
RETURNS TEXT
AS $$
DECLARE
denominator_id TEXT;
categories TEXT[];
data_table TEXT;
geom_table TEXT;
colname TEXT;
data_geomref_colname TEXT;
geom_geomref_colname TEXT;
geom_colname TEXT;
category_val TEXT;
category_share NUMERIC;
BEGIN
IF boundary_id IS NULL THEN
-- TODO we should determine best boundary for this geom
boundary_id := 'us.census.tiger.census_tract';
EXECUTE
$query$
SELECT numer_colname, numer_geomref_colname, numer_tablename,
geom_geomref_colname, geom_colname, geom_tablename
FROM observatory.obs_meta
WHERE (geom_id = $1 OR ($1 = ''))
AND numer_id = $2
AND (numer_timespan = $3 OR ($3 = ''))
ORDER BY geom_weight DESC, numer_timespan DESC
LIMIT 1
$query$
INTO colname, data_geomref_colname, data_table,
geom_geomref_colname, geom_colname, geom_table
USING COALESCE(boundary_id, ''), category_id, COALESCE(time_span, '');
IF ST_GeometryType(geom) = 'ST_Point' THEN
EXECUTE format(
'SELECT data.%I
FROM observatory.%I data, observatory.%I geom
WHERE data.%I = geom.%I
AND ST_WITHIN(%L, geom.%I) ',
colname, data_table, geom_table, data_geomref_colname,
geom_geomref_colname, geom, geom_colname)
INTO category_val;
ELSE
-- favor the category with the most area
EXECUTE format(
'SELECT data.%I category, SUM(overlap_fraction) category_share
FROM observatory.%I data, (
SELECT ST_Area(
ST_Intersection(%L, a.%I)
) / ST_Area(%L) AS overlap_fraction, a.%I geomref
FROM observatory.%I as a
WHERE %L && a.%I) _overlaps
WHERE data.%I = _overlaps.geomref
GROUP BY category
ORDER BY SUM(overlap_fraction) DESC
LIMIT 1',
colname, data_table,
geom, geom_colname, geom, geom_geomref_colname,
geom_table, geom, geom_colname, data_geomref_colname)
INTO category_val, category_share;
END IF;
IF time_span IS NULL THEN
-- TODO we should determine latest timespan for this measure
time_span := '2010 - 2014';
END IF;
EXECUTE '
SELECT ARRAY_AGG(val) FROM (SELECT (cdb_observatory._OBS_GetCategories($1, $2, $3, $4))->>''category'' val LIMIT 1) b
'
INTO categories
USING geom, ARRAY[category_id], boundary_id, time_span;
RETURN (categories)[1];
RETURN category_val;
END;
$$ LANGUAGE plpgsql;
@@ -464,7 +624,7 @@ $$ LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetUSCensusMeasure(
geom geometry(Geometry, 4326),
name TEXT,
normalize TEXT DEFAULT 'area',
normalize TEXT DEFAULT NULL,
boundary_id TEXT DEFAULT NULL,
time_span TEXT DEFAULT NULL
)
@@ -530,7 +690,7 @@ $$ LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetPopulation(
geom geometry(Geometry, 4326),
normalize TEXT DEFAULT 'area',
normalize TEXT DEFAULT NULL,
boundary_id TEXT DEFAULT NULL,
time_span TEXT DEFAULT NULL
)
@@ -796,7 +956,7 @@ BEGIN
IF geom_table_name IS NULL
THEN
RAISE NOTICE 'Point % is outside of the data region', ST_AsText(geom);
--raise notice 'Point % is outside of the data region', ST_AsText(geom);
RETURN QUERY SELECT '{}'::text[], '{}'::text[];
RETURN;
END IF;
@@ -810,7 +970,7 @@ BEGIN
IF data_table_info IS NULL
THEN
RAISE NOTICE 'No data table found for this location';
--raise notice 'No data table found for this location';
RETURN QUERY SELECT NULL::json;
RETURN;
END IF;
@@ -825,7 +985,7 @@ BEGIN
IF geoid IS NULL
THEN
RAISE NOTICE 'No geometry id for this location';
--raise notice 'No geometry id for this location';
RETURN QUERY SELECT NULL::json;
RETURN;
END IF;

20
src/pg/sql/44_observatory_geometries.sql
View File

@@ -64,11 +64,11 @@ BEGIN
-- if no tables are found, raise notice and return null
IF target_table IS NULL
THEN
RAISE NOTICE 'No boundaries found for ''%'' in ''%''', ST_AsText(geom), boundary_id;
--RAISE NOTICE 'No boundaries found for ''%'' in ''%''', ST_AsText(geom), boundary_id;
RETURN NULL::geometry;
END IF;
RAISE NOTICE 'target_table: %', target_table;
--RAISE NOTICE 'target_table: %', target_table;
-- return the first boundary in intersections
EXECUTE format(
@@ -143,7 +143,7 @@ BEGIN
-- if no tables are found, raise notice and return null
IF target_table IS NULL
THEN
RAISE NOTICE 'Warning: No boundaries found for ''%''', boundary_id;
--RAISE NOTICE 'Warning: No boundaries found for ''%''', boundary_id;
RETURN NULL::text;
END IF;
@@ -159,7 +159,7 @@ BEGIN
, target_table)
INTO geoid_colname;
RAISE NOTICE 'target_table: %, geoid_colname: %', target_table, geoid_colname;
--RAISE NOTICE 'target_table: %, geoid_colname: %', target_table, geoid_colname;
-- return geometry id column value
EXECUTE format(
@@ -212,11 +212,11 @@ BEGIN
SELECT * INTO geoid_colname, target_table, geom_colname
FROM cdb_observatory._OBS_GetGeometryMetadata(boundary_id);
RAISE NOTICE '%', target_table;
--RAISE NOTICE '%', target_table;
IF target_table IS NULL
THEN
RAISE NOTICE 'No geometries found';
--RAISE NOTICE 'No geometries found';
RETURN NULL::geometry;
END IF;
@@ -272,12 +272,12 @@ BEGIN
-- if no tables are found, raise notice and return null
IF target_table IS NULL
THEN
RAISE NOTICE 'No boundaries found for bounding box ''%'' in ''%''', ST_AsText(geom), boundary_id;
--RAISE NOTICE 'No boundaries found for bounding box ''%'' in ''%''', ST_AsText(geom), boundary_id;
RETURN QUERY SELECT NULL::geometry, NULL::text;
RETURN;
END IF;
RAISE NOTICE 'target_table: %', target_table;
--RAISE NOTICE 'target_table: %', target_table;
-- return first boundary in intersections
RETURN QUERY
@@ -418,12 +418,12 @@ BEGIN
-- if no tables are found, raise notice and return null
IF target_table IS NULL
THEN
RAISE NOTICE 'No boundaries found for bounding box ''%'' in ''%''', ST_AsText(geom), boundary_id;
--RAISE NOTICE 'No boundaries found for bounding box ''%'' in ''%''', ST_AsText(geom), boundary_id;
RETURN QUERY SELECT NULL::geometry, NULL::text;
RETURN;
END IF;
RAISE NOTICE 'target_table: %', target_table;
--RAISE NOTICE 'target_table: %', target_table;
-- return first boundary in intersections
RETURN QUERY

9
src/pg/test/expected/40_observatory_utility_test.out
View File

@@ -9,21 +9,12 @@ t
_obs_geomtable_with_null_response
t
(1 row)
test_get_obs_column_with_geoid_and_census_1|test_get_obs_column_with_geoid_and_census_2
t|t
(1 row)
obs_getcolumndata_missing_measure
t
(1 row)
_obs_buildsnapshotquery_test_1
t
(1 row)
_obs_buildsnapshotquery_test_2
t
(1 row)
_obs_getrelatedcolumn_test
t
(1 row)
_obs_standardizemeasurename_test
t
(1 row)

3
src/pg/test/expected/41_observatory_augmentation_test.out
View File

@@ -1,5 +1,4 @@
\i test/fixtures/load_fixtures.sql
SET client_min_messages TO WARNING;
\pset format unaligned
\set ECHO none
obs_getdemographicsnapshot_test_no_returns
t

3
src/pg/test/expected/42_observatory_exploration_test.out
View File

@@ -1,5 +1,4 @@
\i test/fixtures/load_fixtures.sql
SET client_min_messages TO WARNING;
\pset format unaligned
\set ECHO none
_obs_searchtables_tables_match|_obs_searchtables_timespan_matches
t|t

3
src/pg/test/expected/44_observatory_geometries_test.out
View File

@@ -1,7 +1,4 @@
\pset format unaligned
\set ECHO all
\i test/fixtures/load_fixtures.sql
SET client_min_messages TO WARNING;
\set ECHO none
obs_getboundary_cartodb_census_tract
t

2
src/pg/test/fixtures/drop_fixtures.sql vendored
View File

@@ -20,3 +20,5 @@ DROP TABLE IF EXISTS observatory.obs_6c1309a64d8f3e6986061f4d1ca7b57743e75e74;
DROP TABLE IF EXISTS observatory.obs_d39f7fe5959891c8296490d83c22ded31c54af13;
DROP TABLE IF EXISTS observatory.obs_144e8b4f906885b2e057ac4842644a553ae49c6e;
DROP TABLE IF EXISTS observatory.obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308;
DROP TABLE IF EXISTS observatory.obs_meta;

20633
src/pg/test/fixtures/load_fixtures.sql vendored
View File

File diff suppressed because one or more lines are too long

36
src/pg/test/sql/40_observatory_utility_test.sql
View File

@@ -1,6 +1,8 @@
\pset format unaligned
\set ECHO all
\i test/fixtures/load_fixtures.sql
SET client_min_messages TO WARNING;
\set ECHO none
-- OBS_GeomTable
-- get table with known geometry_id
@@ -27,29 +29,6 @@ SELECT
-- 'us.census.tiger.census_tract'
-- );
WITH result as (
SELECT
array_agg(a) expected from cdb_observatory._OBS_GetColumnData(
'us.census.tiger.census_tract',
Array['us.census.spielman_singleton_segments.X55', 'us.census.acs.B01003001'],
'2010 - 2014') a
)
select
(expected)[1]::text = '{"colname":"x55","tablename":"obs_65f29658e096ca1485bf683f65fdbc9f05ec3c5d","aggregate":null,"name":"Spielman-Singleton Segments: 55 Clusters","type":"Text","description":"Sociodemographic classes from Spielman and Singleton 2015, 55 clusters","boundary_id":"us.census.tiger.census_tract"}' as test_get_obs_column_with_geoid_and_census_1,
(expected)[2]::text = '{"colname":"total_pop","tablename":"obs_b393b5b88c6adda634b2071a8005b03c551b609a","aggregate":"sum","name":"Total Population","type":"Numeric","description":"The total number of all people living in a given geographic area. This is a very useful catch-all denominator when calculating rates.","boundary_id":"us.census.tiger.census_tract"}' as test_get_obs_column_with_geoid_and_census_2
from result;
-- should be null-valued
WITH result as (
SELECT
array_agg(a) expected from cdb_observatory._OBS_GetColumnData(
'us.census.tiger.census_tract',
Array['us.census.tiger.baloney'],
'2010 - 2014') a
)
select expected is null as OBS_GetColumnData_missing_measure
from result;
-- OBS_BuildSnapshotQuery
-- Should give back: SELECT vals[1] As total_pop, vals[2] As male_pop, vals[3] As female_pop, vals[4] As median_age
SELECT
@@ -63,19 +42,8 @@ SELECT
Array['mandarin_orange']
) = 'SELECT vals[1] As mandarin_orange' As _OBS_BuildSnapshotQuery_test_2;
SELECT cdb_observatory._OBS_GetRelatedColumn(
Array[
'es.ine.t3_1',
'us.census.acs.B01003001',
'us.census.acs.B01001002'
],
'denominator'
) = '{es.ine.t1_1,NULL,us.census.acs.B01003001}' As _OBS_GetRelatedColumn_test;
-- should give back a standardized measure name
SELECT cdb_observatory._OBS_StandardizeMeasureName('test 343 %% 2 qqq }}{{}}') = 'test_343_2_qqq' As _OBS_StandardizeMeasureName_test;
SELECT cdb_observatory.OBS_DumpVersion()
IS NOT NULL AS OBS_DumpVersion_notnull;
\i test/fixtures/drop_fixtures.sql

10
src/pg/test/sql/41_observatory_augmentation_test.sql
View File

@@ -1,6 +1,6 @@
\i test/fixtures/load_fixtures.sql
\pset format unaligned
\set ECHO none
SET client_min_messages TO WARNING;
--
WITH result as(
@@ -131,7 +131,7 @@ WITH result as (
-- Point-based OBS_GetMeasure with zillow
SELECT abs(OBS_GetMeasure_zhvi_point - 583600) / 583600 < 0.001 AS OBS_GetMeasure_zhvi_point_test FROM cdb_observatory.OBS_GetMeasure(
ST_SetSRID(ST_Point(-73.94602417945862, 40.6768220087458), 4326),
'us.zillow.AllHomes_Zhvi', 'area', 'us.census.tiger.zcta5', '2014-01'
'us.zillow.AllHomes_Zhvi', null, 'us.census.tiger.zcta5', '2014-01'
) As t(OBS_GetMeasure_zhvi_point);
-- Point-based OBS_GetMeasure with zillow default to latest
@@ -172,7 +172,7 @@ SELECT cdb_observatory.OBS_GetCategory(
-- Poly-based OBS_GetCategory
SELECT cdb_observatory.OBS_GetCategory(
cdb_observatory._TestArea(), 'us.census.spielman_singleton_segments.X10') = 'Low income, mix of minorities' As obs_getcategory_polygon;
cdb_observatory._TestArea(), 'us.census.spielman_singleton_segments.X10') = 'Wealthy, urban without Kids' As obs_getcategory_polygon;
-- Point-based OBS_GetPopulation, default normalization (area)
SELECT (abs(OBS_GetPopulation - 10923.093200390833950) / 10923.093200390833950) < 0.001 As OBS_GetPopulation FROM
@@ -201,7 +201,7 @@ SELECT cdb_observatory.OBS_GetUSCensusCategory(
-- Area-based OBS_GetUSCensusCategory
SELECT cdb_observatory.OBS_GetUSCensusCategory(
cdb_observatory._testarea(), 'Spielman-Singleton Segments: 10 Clusters') = 'Low income, mix of minorities' As OBS_GetUSCensusCategory_polygon;
cdb_observatory._testarea(), 'Spielman-Singleton Segments: 10 Clusters') = 'Wealthy, urban without Kids' As OBS_GetUSCensusCategory_polygon;
-- OBS_GetMeasureById tests
@@ -236,5 +236,3 @@ SELECT cdb_observatory.OBS_GetMeasureById(
'us.census.tiger.block_group',
'2010 - 2014'
) IS NULL As OBS_GetMeasureById_nulls;
\i test/fixtures/drop_fixtures.sql

5
src/pg/test/sql/42_observatory_exploration_test.sql
View File

@@ -1,5 +1,6 @@
\i test/fixtures/load_fixtures.sql
\pset format unaligned
\set ECHO none
SET client_min_messages TO WARNING;
-- set up variables for use in testing
@@ -32,5 +33,3 @@ SELECT COUNT(*) > 0 AS _OBS_GetAvailableBoundariesExist
FROM cdb_observatory.OBS_GetAvailableBoundaries(
cdb_observatory._TestPoint()
) AS t(boundary_id, description, time_span, tablename);
\i test/fixtures/drop_fixtures.sql

4
src/pg/test/sql/44_observatory_geometries_test.sql
View File

@@ -1,6 +1,6 @@
\pset format unaligned
\set ECHO all
\i test/fixtures/load_fixtures.sql
\set ECHO none
SET client_min_messages TO WARNING;
-- set up variables for use in testing