cartodb4/observatory-extension
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/develop' into more-tests-for-boundaries

Browse Source
This commit is contained in:
John Krauss
2016-05-20 18:02:32 -04:00
parent b5f4a617e8 d189bbe3c7
commit 73dfba2d4b
25 changed files with 17557 additions and 15465 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.gitignore vendored
View File

@@ -2,3 +2,5 @@
src/pg/observatory--current--dev.sql
src/pg/observatory--dev--current.sql
src/pg/observatory--dev.sql
venv
*.pyc

1
doc/API.md
View File

@@ -8,3 +8,4 @@ This file is for reference purposes only. It is intended for tracking the Data O
* [Boundary Functions](boundary_functions.md)
* [Discovery Functions](discovery_functions.md)
* [Glossary](glossary.md)
* [License](license.md)

25
doc/boundary_functions.md
View File

@@ -1,6 +1,8 @@
# Boundary Functions
Use the following functions to retrieve [Boundary](/cartodb-platform/dataobservatory/overview/#boundary-data) data. Data ranges from small areas (e.g. US Census Block Groups) to large areas (e.g. Countries). You can access boundaries by point location lookup, bounding box lookup, direct ID access and several other methods described below.
Use the following functions to retrieve [Boundary](/cartodb-platform/data/overview/#boundary-data) data. Data ranges from small areas (e.g. US Census Block Groups) to large areas (e.g. Countries). You can access boundaries by point location lookup, bounding box lookup, direct ID access and several other methods described below.
You can [access](/cartodb-platform/data/accessing/#accessing-the-data-observatory) boundaries through the CartoDB Editor. The same methods will work if you are using the CartoDB Platform to develop your application. We [encourage you](/cartodb-platform/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
## OBS_GetBoundariesByGeometry(polygon geometry, geometry_id text)
@@ -28,10 +30,10 @@ If geometries are not found for the requested `polygon`, `geometry_id`, `timespa
#### Example
Insert all Census Tracts from Lower Manhattan and nearby areas within the supplied bounding box to a table named `manhattan_census_tracts` which has columns `the_geom` (geometry) and `geoid` (text).
Insert all Census Tracts from Lower Manhattan and nearby areas within the supplied bounding box to a table named `manhattan_census_tracts` which has columns `the_geom` (geometry) and `geom_refs` (text).
```sql
INSERT INTO manhattan_census_tracts(the_geom, geoid)
INSERT INTO manhattan_census_tracts(the_geom, geom_refs)
SELECT *
FROM OBS_GetBoundariesByGeometry(
ST_MakeEnvelope(-74.0251922607,40.6945658517,
@@ -67,14 +69,14 @@ Column Name | Description
the_geom | a point geometry on a boundary (e.g., a point that lies on a US Census tract)
geom_refs| a string identifier for the geometry (e.g., the geoid of a US Census tract)
If geometries are not found for the requested geometry, `geometry_id`, `timespan`, or `overlap_type`, then null values are returned.
If geometries are not found for the requested geometry, `geometry_id`, `timespan`, or `overlap_type`, then NULL values are returned.
#### Example
Insert points that lie on Census Tracts from Lower Manhattan and nearby areas within the supplied bounding box to a table named `manhattan_census_tracts` which has columns `the_geom` (geometry) and `geoid` (text).
Insert points that lie on Census Tracts from Lower Manhattan and nearby areas within the supplied bounding box to a table named `manhattan_tract_points` which has columns `the_geom` (geometry) and `geom_refs` (text).
```sql
INSERT INTO manhattan_census_tract_points(the_geom, geoid)
INSERT INTO manhattan_tract_points (the_geom, geom_refs)
SELECT *
FROM OBS_GetPointsByGeometry(
ST_MakeEnvelope(-74.0251922607,40.6945658517,
@@ -175,9 +177,10 @@ geom | a WGS84 polygon geometry
#### Example
Use a table of `geometry_id`s (e.g., geoid from the U.S. Census) to select the unique boundaries that they correspond to.
Use a table of `geometry_id`s (e.g., geoid from the U.S. Census) to select the unique boundaries that they correspond to and insert into a table called, `overlapping_polygons`. This is a useful method for creating new choropleths of aggregate data.
```SQL
INSERT INTO overlapping_polygons (the_geom, geometry_id, point_count)
SELECT
OBS_GetBoundaryById(geometry_id, 'us.census.tiger.county') As the_geom,
geometry_id,
@@ -213,10 +216,10 @@ If geometries are not found for the requested point and radius, `geometry_id`, `
#### Example
Insert into table `denver_census_tracts` the census tract boundaries and geoids of census tracts which intersect within 10 miles of downtown Denver, Colorado.
Insert into table `denver_census_tracts` the census tract boundaries and geom_refs of census tracts which intersect within 10 miles of downtown Denver, Colorado.
```sql
INSERT INTO denver_census_tracts(the_geom, geoid)
INSERT INTO denver_census_tracts(the_geom, geom_refs)
SELECT *
FROM OBS_GetBoundariesByPointAndRadius(
CDB_LatLng(39.7392, -104.9903), -- Denver, Colorado
@@ -255,10 +258,10 @@ If geometries are not found for the requested point and radius, `geometry_id`, `
#### Example
Insert into table `denver_census_tracts` points on US census tracts and their corresponding geoids for census tracts which intersect within 10 miles of downtown Denver, Colorado, USA.
Insert into table `denver_tract_points` points on US census tracts and their corresponding geoids for census tracts which intersect within 10 miles of downtown Denver, Colorado, USA.
```sql
INSERT INTO denver_census_tracts(the_geom, geoid)
INSERT INTO denver_tract_points(the_geom, geom_refs)
SELECT *
FROM OBS_GetPointsByPointAndRadius(
CDB_LatLng(39.7392, -104.9903), -- Denver, Colorado

14
doc/discovery_functions.md
View File

@@ -1,6 +1,6 @@
# Discovery Functions
If you are using the [discovery methods](/cartodb-platform/dataobservatory/overview/#discovery-methods) from the Data Observatory, use the following functions to retrieve [boundary](/cartodb-platform/dataobservatory/overview/#boundary-data) and [measures](/cartodb-platform/dataobservatory/overview/#measures-data) data.
If you are using the [discovery methods](/cartodb-platform/data/overview/#discovery-methods) from the Data Observatory, use the following functions to retrieve [boundary](/cartodb-platform/data/overview/#boundary-data) and [measures](/cartodb-platform/data/overview/#measures-data) data.
## OBS_Search(search_term)
@@ -19,16 +19,16 @@ A TABLE containing the following properties
Key | Description
--- | ---
measure_id | the unique id of the measure for use with the ```OBS_GetMeasure``` function
id | the unique id of the measure for use with the ```OBS_GetMeasure``` function
name | the human readable name of the measure
description | a brief description of the measure
aggregate_type | **sum** are raw count values, **median** are statistical medians, **average** are statistical averages, **undefined** other (e.g. an index value)
sources | where the data came from (e.g. US Census Bureau)
aggregate | **sum** are raw count values, **median** are statistical medians, **average** are statistical averages, **undefined** other (e.g. an index value)
source | where the data came from (e.g. US Census Bureau)
#### Example
```SQL
SELECT * FROM OBS_Search('inequality')
SELECT * FROM OBS_Search('home value')
```
## OBS_GetAvailableBoundaries(point_geometry)
@@ -47,9 +47,9 @@ A TABLE containing the following properties
Key | Description
--- | ---
boundary_id | a boundary identifier from the [boundary ID glossary](/cartodb-platform/dataobservatory/glossary/#boundary-ids)
boundary_id | a boundary identifier from the [boundary ID glossary](/cartodb-platform/data/glossary/#boundary-ids)
description | a brief description of the boundary dataset
timespan | the timespan attached the boundary. this does not mean that the boundary is invalid outside of the timespan, but is the explicit timespan published with the geometry.
time_span | the timespan attached the boundary. this does not mean that the boundary is invalid outside of the timespan, but is the explicit timespan published with the geometry.
#### Example

16
doc/glossary.md
View File

@@ -1,6 +1,6 @@
# Glossary
A list of boundary ids and measure_names for Data Observatory functions. For US based boundaries, the Shoreline Clipped version provides a high-quality shoreline clipping for mapping uses.
A list of boundary ids and measure_names for Data Observatory functions. For US based boundaries, the Shoreline Clipped version provides a high-quality shoreline clipping for mapping uses.
## Boundary IDs
@@ -18,12 +18,12 @@ US Census Blocks | us.census.tiger.block | us.census.tiger.block_clipped
US Census Block Groups | us.census.tiger.block_group | us.census.tiger.block_group_clipped
US Census PUMAs | us.census.tiger.puma | us.census.tiger.puma_clipped
US Incorporated Places | us.census.tiger.place | us.census.tiger.place_clipped
ES Sección Censal | es.ine.geom |
Regions (First-level Administrative) | whosonfirst.wof_region_geom |
Continents | whosonfirst.wof_continent_geom |
Countries | whosonfirst.wof_country_geom |
Marine Areas | whosonfirst.wof_marinearea_geom |
Disputed Areas | whosonfirst.wof_disputed_geom |
ES Sección Censal | es.ine.geom | none
Regions (First-level Administrative) | whosonfirst.wof_region_geom | none
Continents | whosonfirst.wof_continent_geom | none
Countries | whosonfirst.wof_country_geom | none
Marine Areas | whosonfirst.wof_marinearea_geom | none
Disputed Areas | whosonfirst.wof_disputed_geom | none
@@ -67,7 +67,7 @@ Number of workers with a commute between 25 and 29 minutes | The number of w
Number of workers with a commute between 30 and 34 minutes | The number of workers over the age of 16 who do not work from home and commute in between 30 and 34 minutes in a geographic area.
Number of workers with a commute between 45 and 59 minutes | The number of workers over the age of 16 who do not work from home and commute in between 45 and 59 minutes in a geographic area.
Children under 18 Years of Age | The number of people within each geography who are under 18 years of age.
Households | A count of the number of households in each geography. A household consists of one or more people who live in the same dwelling and also share at meals or living accommodation, and may consist of a single family or some other grouping of people.
Households | A count of the number of households in each geography. A household consists of one or more people who live in the same dwelling and also share at meals or living accommodation, and may consist of a single family or some other grouping of people.
Population 15 Years and Over | The number of people in a geographic area who are over the age of 15. This is used mostly as a denominator of marital status.
Never Married | The number of people in a geographic area who have never been married.
Currently married | The number of people in a geographic area who are currently married.

18
doc/license.md Normal file
View File

@@ -0,0 +1,18 @@
# License
The Data Observatory is a collection of various sources of data with varying licenses. We have worked hard to find you data that will work for the broadest set of use-cases. For competency, please still review the terms for any dataset you use and respect the rights of the owners for each dataset. The following third-party data sources are used in the Data Observatory, and we have included the links to the terms governing their use.
Name | Terms link
-------|---------
ACS | [https://www.usa.gov/government-works](https://www.usa.gov/government-works)
TIGER | [https://www.usa.gov/government-works](https://www.usa.gov/government-works)
Zillow Home Value Index | This data is "Aggregate Data", per the Zillow Terms of Use<br /><br />[http://www.zillow.com/corp/Terms.htm](http://www.zillow.com/corp/Terms.htm)
Who's on First | [http://whosonfirst.mapzen.com#License](http://whosonfirst.mapzen.com#License)
GeoNames | [http://www.geonames.org/](http://www.geonames.org/)
GeoPlanet | [https://developer.yahoo.com/geo/geoplanet/](https://developer.yahoo.com/geo/geoplanet/)
Natural Earth | [http://www.naturalearthdata.com/about/terms-of-use/](http://www.naturalearthdata.com/about/terms-of-use/)
Quattroshapes | [https://github.com/foursquare/quattroshapes/blob/master/LICENSE.md](https://github.com/foursquare/quattroshapes/blob/master/LICENSE.md)
Zetashapes | [http://zetashapes.com/license](http://zetashapes.com/license)
Spielman & Singleton | [https://www.openicpsr.org/repoEntity/show/41329](https://www.openicpsr.org/repoEntity/show/41329)
Instituto Nacional de Estadistica | [http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout](http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout)

34
doc/measures_functions.md
View File

@@ -1,21 +1,21 @@
# Measures Functions
[Data Observatory Measures](/cartodb-platform/dataobservatory/overview/#measures-methods) are the numerical location data you can access. The measure Functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (Point or Polygon) and receiving back a measure (an absolute value) for that location.
[Data Observatory Measures](/cartodb-platform/data/overview/#measures-methods) are the numerical location data you can access. The measure functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (point or polygon) and receiving back a measure (an absolute value) for that location.
There are hundreds of Measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf).
There are hundreds of measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf).
We show here how you can access Measures through the CartoDB Editor. The same methods will work if you are using the CartoDB Platform to develop your application. We encourage you to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
You can [access](/cartodb-platform/data/accessing/#accessing-the-data-observatory) measures through the CartoDB Editor. The same methods will work if you are using the CartoDB Platform to develop your application. We [encourage you](/cartodb-platform/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
## OBS_GetUSCensusMeasure(point geometry, measure_name text);
## OBS_GetUSCensusMeasure(point geometry, measure_name text)
The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure based on a subset of the US Census variables at a point location. The ```OBS_GetUSCensusMeasure``` function is limited to only a subset of all Measures that are available in the Data Observatory, to access the full list, use the ```OBS_GetMeasure``` function below.
The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure based on a subset of the US Census variables at a point location. The ```OBS_GetUSCensusMeasure``` function is limited to only a subset of all measures that are available in the Data Observatory, to access the full list, use measure IDs with the ```OBS_GetMeasure``` function below.
#### Arguments
Name |Description
--- | ---
point | a WGS84 point geometry (the_geom)
measure_name | a human readable name of a US Census variable. The list of measure_names is [available in the glossary](/cartodb-platform/dataobservatory/glossary/#obsgetuscensusmeasure-names-table).
measure_name | a human readable name of a US Census variable. The list of measure_names is [available in the glossary](/cartodb-platform/data/glossary/#obsgetuscensusmeasure-names-table).
normalize | for measures that are are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://cartodb.github.io/bigmetadata/index.html) (optional)
boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
time_span | time span of interest (e.g., 2010 - 2014)
@@ -33,11 +33,11 @@ value | the raw or normalized measure
Add a measure to an empty numeric column based on point locations in your table.
```SQL
UPDATE tablename
SET local_male_population = OBS_GetUSCensusMeasure(the_geom, 'Male Population')
UPDATE tablename
SET total_population = OBS_GetUSCensusMeasure(the_geom, 'Total Population')
```
## OBS_GetUSCensusMeasure(polygon geometry, measure_name text);
## OBS_GetUSCensusMeasure(polygon geometry, measure_name text)
The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure based on a subset of the US Census variables within a given polygon. The ```OBS_GetUSCensusMeasure``` function is limited to only a subset of all measures that are available in the Data Observatory, to access the full list, use the ```OBS_GetUSCensusMeasure``` function below.
@@ -46,7 +46,7 @@ The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure
Name |Description
--- | ---
polygon | a WGS84 polygon geometry (the_geom)
measure_name | a human readable string name of a US Census variable. The list of measure_names is [available in the glossary](/cartodb-platform/dataobservatory/glossary/#obsgetuscensusmeasure-names-table).
measure_name | a human readable string name of a US Census variable. The list of measure_names is [available in the glossary](/cartodb-platform/data/glossary/#obsgetuscensusmeasure-names-table).
normalize | for measures that are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf) (optional)
boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
time_span | time span of interest (e.g., 2010 - 2014)
@@ -68,7 +68,7 @@ UPDATE tablename
SET local_male_population = OBS_GetUSCensusMeasure(the_geom, 'Male Population')
```
## OBS_GetMeasure(point geometry, measure_id text);
## OBS_GetMeasure(point geometry, measure_id text)
The ```OBS_GetMeasure(point, measure_id)``` function returns any Data Observatory measure at a point location. You can browse all available Measures in the [Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf)).
@@ -96,10 +96,10 @@ Add a measure to an empty numeric column based on point locations in your table
```SQL
UPDATE tablename
SET local_male_population = OBS_GetMeasure(the_geom, 'us.census.acs.B08134006')
SET median_home_value_sqft = OBS_GetMeasure(the_geom, 'us.zillow.AllHomes_MedianValuePerSqft')
```
## OBS_GetMeasure(polygon geometry, measure_id text);
## OBS_GetMeasure(polygon geometry, measure_id text)
The ```OBS_GetMeasure(polygon, measure_id)``` function returns any Data Observatory measure calculated within a polygon.
@@ -126,15 +126,15 @@ value | the raw or normalized measure
Add a measure to an empty column based on polygons in your table
```SQL
UPDATE tablename
SET local_male_population = OBS_GetMeasure(the_geom, 'us.census.acs.B08134006')
UPDATE tablename
SET household_count = OBS_GetMeasure(the_geom, 'us.census.acs.B11001001')
```
#### Errors
* If an unrecognized normalization type is input, raise an error: `'Only valid inputs for "normalize" are "area" (default) and "denominator".`
## OBS_GetCategory(point geometry, category_id text);
## OBS_GetCategory(point geometry, category_id text)
The ```OBS_GetCategory(point, category_id)``` function returns any Data Observatory Category value at a point location. The Categories available are currently limited to Segmentation categories. See the Segmentation section of the [Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf) for more detail.
@@ -158,6 +158,6 @@ value | a text based category found at the supplied point
Add the Category to an empty column text column based on point locations in your table
```SQL
UPDATE tablename
UPDATE tablename
SET segmentation = OBS_GetCategory(the_geom, 'us.census.spielman_singleton_segments.X55')
```

866
release/observatory--0.0.1.sql Normal file
View File

@@ -0,0 +1,866 @@
--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
\echo Use "CREATE EXTENSION observatory" to load this file. \quit
-- Version number of the extension release
CREATE OR REPLACE FUNCTION cdb_observatory_version()
RETURNS text AS $$
SELECT '0.0.1'::text;
$$ language 'sql' STABLE STRICT;
-- Internal identifier of the installed extension instence
-- e.g. 'dev' for current development version
CREATE OR REPLACE FUNCTION _cdb_observatory_internal_version()
RETURNS text AS $$
SELECT installed_version FROM pg_available_extensions where name='observatory' and pg_available_extensions IS NOT NULL;
$$ language 'sql' STABLE STRICT;
-- Returns the table name with geoms for the given geometry_id
-- TODO probably needs to take in the column_id array to get the relevant
-- table where there is multiple sources for a column from multiple
-- geometries.
CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GeomTable(
geom geometry,
geometry_id text
)
RETURNS TEXT
AS $$
DECLARE
result text;
BEGIN
EXECUTE '
SELECT tablename FROM observatory.OBS_table
WHERE id IN (
SELECT table_id
FROM observatory.OBS_table tab,
observatory.OBS_column_table coltable,
observatory.OBS_column col
WHERE type ILIKE ''geometry''
AND coltable.column_id = col.id
AND coltable.table_id = tab.id
AND col.id = $1
)
'
USING geometry_id, geom
INTO result;
return result;
END;
$$ LANGUAGE plpgsql;
-- A type for use with the OBS_GetColumnData function
CREATE TYPE cdb_observatory.OBS_ColumnData AS (colname text, tablename text, aggregate text);
-- A function that gets the column data for multiple columns
-- Old: OBS_GetColumnData
CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetColumnData(
geometry_id text,
column_ids text[],
timespan text
)
RETURNS cdb_observatory.OBS_ColumnData[]
AS $$
DECLARE
result cdb_observatory.OBS_ColumnData[];
BEGIN
EXECUTE '
WITH geomref AS (
SELECT t.table_id id
FROM observatory.OBS_column_to_column c2c, observatory.OBS_column_table t
WHERE c2c.reltype = ''geom_ref''
AND c2c.target_id = $1
AND c2c.source_id = t.column_id
),
column_ids as (
select row_number() over () as no, a.column_id as column_id from (select unnest($2) as column_id) a
)
SELECT array_agg(ROW(colname, tablename, aggregate)::cdb_observatory.OBS_ColumnData order by column_ids.no)
FROM column_ids, observatory.OBS_column c, observatory.OBS_column_table ct, observatory.OBS_table t
WHERE column_ids.column_id = c.id
AND c.id = ct.column_id
AND t.id = ct.table_id
AND t.timespan = $3
AND t.id in (SELECT id FROM geomref)
'
USING geometry_id, column_ids, timespan
INTO result;
RETURN result;
END;
$$ LANGUAGE plpgsql;
--Gets the column id for a census variable given a human readable version of it
-- Old: OBS_LOOKUP_CENSUS_HUMAN
CREATE OR REPLACE FUNCTION cdb_observatory._OBS_LookupCensusHuman(
column_names text[],
-- TODO: change variable name table_name to table_id
table_name text DEFAULT '"us.census.acs".extract_block_group_5yr_2013_69b156927c'
)
RETURNS text[] as $$
DECLARE
column_id text;
result text;
BEGIN
EXECUTE format('
WITH col_names AS (
select row_number() over() as no, a.column_name as column_name from(
select unnest($1) as column_name
) a
)
select array_agg(column_id order by col_names.no)
FROM observatory.OBS_column_table,col_names
where colname = col_names.column_name
and table_id = %L limit 1
', table_name)
INTO result
using column_names;
RETURN result;
END
$$ LANGUAGE plpgsql;
--Test point cause Stuart always seems to make random points in the water
CREATE OR REPLACE FUNCTION cdb_observatory._TestPoint()
RETURNS geometry
AS $$
BEGIN
-- new york city
RETURN CDB_LatLng(40.704512, -73.936669);
END;
$$ LANGUAGE plpgsql;
--Test polygon cause Stuart always seems to make random points in the water
-- TODO: remove as it's not used anywhere?
CREATE OR REPLACE FUNCTION cdb_observatory._TestArea()
RETURNS geometry
AS $$
BEGIN
-- Buffer NYC point by 500 meters
RETURN ST_Buffer(cdb_observatory._TestPoint()::geography, 500)::geometry;
END;
$$ LANGUAGE plpgsql;
--Used to expand a column based response to a table based one. Give it the desired
--columns and it will return a partial query for rolling them out to a table.
CREATE OR REPLACE FUNCTION cdb_observatory._OBS_BuildSnapshotQuery(names text[])
RETURNS TEXT
AS $$
DECLARE
q text;
i numeric;
BEGIN
q := 'SELECT ';
FOR i IN 1..array_upper(names,1)
LOOP
q = q || format(' vals[%s] As %I', i, names[i]);
IF i < array_upper(names, 1) THEN
q= q || ',';
END IF;
END LOOP;
RETURN q;
END;
$$ LANGUAGE plpgsql;
--For Longer term Dev
--Break out table definitions to types
--Automate type creation from a script, something like
----CREATE OR REPLACE FUNCTION OBS_Get<%=tag_name%>(geom GEOMETRY)
----RETURNS TABLE(
----<%=get_dimensions_for_tag(tag_name)%>
----AS $$
----DECLARE
----target_cols text[];
----names text[];
----vals NUMERIC[];-
----q text;
----BEGIN
----target_cols := Array[<%=get_dimensions_for_tag(tag_name)%>],
--Functions for augmenting specific tables
--------------------------------------------------------------------------------
-- Creates a table of demographic snapshot
CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetDemographicSnapshot(geom geometry, time_span text default '2009 - 2013', geometry_level text default '"us.census.tiger".block_group')
RETURNS json
AS $$
BEGIN
RETURN row_to_json(cdb_observatory._OBS_GetDemographicSnapshot(geom, time_span, geometry_level));
END;
$$ LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetDemographicSnapshot(geom geometry, time_span text default '2009 - 2013', geometry_level text default '"us.census.tiger".block_group' )
RETURNS TABLE(
total_pop NUMERIC,
male_pop NUMERIC,
female_pop NUMERIC,
median_age NUMERIC,
white_pop NUMERIC,
black_pop NUMERIC,
asian_pop NUMERIC,
hispanic_pop NUMERIC,
amerindian_pop NUMERIC,
other_race_pop NUMERIC,
two_or_more_races_pop NUMERIC,
not_hispanic_pop NUMERIC,
--not_us_citizen_pop NUMERIC,
--workers_16_and_over NUMERIC,
--commuters_by_car_truck_van NUMERIC,
--commuters_drove_alone NUMERIC,
--commuters_by_carpool NUMERIC,
--commuters_by_public_transportation NUMERIC,
--commuters_by_bus NUMERIC,
--commuters_by_subway_or_elevated NUMERIC,
--walked_to_work NUMERIC,
--worked_at_home NUMERIC,
--children NUMERIC, -- TODO we should be able to get this at BG
households NUMERIC,
--population_3_years_over NUMERIC,
--in_school NUMERIC,
--in_grades_1_to_4 NUMERIC,
--in_grades_5_to_8 NUMERIC,
--in_grades_9_to_12 NUMERIC,
--in_undergrad_college NUMERIC,
pop_25_years_over NUMERIC,
high_school_diploma NUMERIC,
less_one_year_college NUMERIC,
one_year_more_college NUMERIC,
associates_degree NUMERIC,
bachelors_degree NUMERIC,
masters_degree NUMERIC,
--pop_5_years_over NUMERIC,
--speak_only_english_at_home NUMERIC,
--speak_spanish_at_home NUMERIC,
--pop_determined_poverty_status NUMERIC,
--poverty NUMERIC,
median_income NUMERIC,
gini_index NUMERIC,
income_per_capita NUMERIC,
housing_units NUMERIC,
vacant_housing_units NUMERIC,
vacant_housing_units_for_rent NUMERIC,
vacant_housing_units_for_sale NUMERIC,
median_rent NUMERIC,
percent_income_spent_on_rent NUMERIC,
owner_occupied_housing_units NUMERIC,
million_dollar_housing_units NUMERIC,
mortgaged_housing_units NUMERIC,
--pop_15_and_over NUMERIC,
--pop_never_married NUMERIC,
--pop_now_married NUMERIC,
--pop_separated NUMERIC,
--pop_widowed NUMERIC,
--pop_divorced NUMERIC,
commuters_16_over NUMERIC,
commute_less_10_mins NUMERIC,
commute_10_14_mins NUMERIC,
commute_15_19_mins NUMERIC,
commute_20_24_mins NUMERIC,
commute_25_29_mins NUMERIC,
commute_30_34_mins NUMERIC,
commute_35_44_mins NUMERIC,
commute_45_59_mins NUMERIC,
commute_60_more_mins NUMERIC,
aggregate_travel_time_to_work NUMERIC,
income_less_10000 NUMERIC,
income_10000_14999 NUMERIC,
income_15000_19999 NUMERIC,
income_20000_24999 NUMERIC,
income_25000_29999 NUMERIC,
income_30000_34999 NUMERIC,
income_35000_39999 NUMERIC,
income_40000_44999 NUMERIC,
income_45000_49999 NUMERIC,
income_50000_59999 NUMERIC,
income_60000_74999 NUMERIC,
income_75000_99999 NUMERIC,
income_100000_124999 NUMERIC,
income_125000_149999 NUMERIC,
income_150000_199999 NUMERIC,
income_200000_or_more NUMERIC,
land_area NUMERIC)
AS $$
DECLARE
target_cols text[];
names text[];
vals NUMERIC[];
q text;
BEGIN
target_cols := Array['total_pop',
'male_pop',
'female_pop',
'median_age',
'white_pop',
'black_pop',
'asian_pop',
'hispanic_pop',
'amerindian_pop',
'other_race_pop',
'two_or_more_races_pop',
'not_hispanic_pop',
--'not_us_citizen_pop',
--'workers_16_and_over',
--'commuters_by_car_truck_van',
--'commuters_drove_alone',
--'commuters_by_carpool',
--'commuters_by_public_transportation',
--'commuters_by_bus',
--'commuters_by_subway_or_elevated',
--'walked_to_work',
--'worked_at_home',
--'children',
'households',
--'population_3_years_over',
--'in_school',
--'in_grades_1_to_4',
--'in_grades_5_to_8',
--'in_grades_9_to_12',
--'in_undergrad_college',
'pop_25_years_over',
'high_school_diploma',
'less_one_year_college',
'one_year_more_college',
'associates_degree',
'bachelors_degree',
'masters_degree',
--'pop_5_years_over',
--'speak_only_english_at_home',
--'speak_spanish_at_home',
--'pop_determined_poverty_status',
--'poverty',
'median_income',
'gini_index',
'income_per_capita',
'housing_units',
'vacant_housing_units',
'vacant_housing_units_for_rent',
'vacant_housing_units_for_sale',
'median_rent',
'percent_income_spent_on_rent',
'owner_occupied_housing_units',
'million_dollar_housing_units',
'mortgaged_housing_units',
--'pop_15_and_over',
--'pop_never_married',
--'pop_now_married',
--'pop_separated',
--'pop_widowed',
--'pop_divorced',
'commuters_16_over',
'commute_less_10_mins',
'commute_10_14_mins',
'commute_15_19_mins',
'commute_20_24_mins',
'commute_25_29_mins',
'commute_30_34_mins',
'commute_35_44_mins',
'commute_45_59_mins',
'commute_60_more_mins',
'aggregate_travel_time_to_work',
'income_less_10000',
'income_10000_14999',
'income_15000_19999',
'income_20000_24999',
'income_25000_29999',
'income_30000_34999',
'income_35000_39999',
'income_40000_44999',
'income_45000_49999',
'income_50000_59999',
'income_60000_74999',
'income_75000_99999',
'income_100000_124999',
'income_125000_149999',
'income_150000_199999',
'income_200000_or_more',
'land_area'];
q := 'WITH a As (
SELECT
dimension As names,
dimension_value As vals
FROM cdb_observatory._OBS_GetCensus($1,$2,$3,$4)
)' ||
cdb_observatory._OBS_BuildSnapshotQuery(target_cols) ||
' FROM a';
RETURN QUERY
EXECUTE
q
USING geom, target_cols, time_span, geometry_level;
RETURN;
END;
$$ LANGUAGE plpgsql;
--Base functions for performing augmentation
----------------------------------------------------------------------------------------
--Returns arrays of values for the given census dimension names for a given
--point or polygon
CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetCensus(
geom geometry,
dimension_names text[],
time_span text DEFAULT '2009 - 2013',
geometry_level text DEFAULT '"us.census.tiger".block_group'
)
RETURNS TABLE(dimension text[], dimension_value NUMERIC[])
AS $$
DECLARE
ids text[];
BEGIN
ids := cdb_observatory._OBS_LookupCensusHuman(dimension_names);
RETURN QUERY
SELECT names, vals FROM cdb_observatory._OBS_Get(geom, ids, time_span, geometry_level);
END;
$$ LANGUAGE plpgsql;
-- Base augmentation fucntion.
CREATE OR REPLACE FUNCTION cdb_observatory._OBS_Get(
geom geometry,
column_ids text[],
time_span text,
geometry_level text
)
RETURNS TABLE(names text[], vals NUMERIC[])
AS $$
DECLARE
results NUMERIC[];
geom_table_name text;
names text[];
query text;
data_table_info cdb_observatory.OBS_ColumnData[];
BEGIN
geom_table_name := cdb_observatory._OBS_GeomTable(geom, geometry_level);
IF geom_table_name IS NULL
THEN
RAISE NOTICE 'Point % is outside of the data region', geom;
RETURN QUERY SELECT '{}'::text[], '{}'::NUMERIC[];
END IF;
data_table_info := cdb_observatory._OBS_GetColumnData(geometry_level,
column_ids,
time_span);
names := (SELECT array_agg((d).colname)
FROM unnest(data_table_info) As d);
IF ST_GeometryType(geom) = 'ST_Point'
THEN
results := cdb_observatory._OBS_GetPoints(geom,
geom_table_name,
data_table_info);
ELSIF ST_GeometryType(geom) IN ('ST_Polygon', 'ST_MultiPolygon')
THEN
results := cdb_observatory._OBS_GetPolygons(geom,
geom_table_name,
data_table_info);
END IF;
IF results IS NULL
THEN
results := Array[]::numeric[];
END IF;
RETURN QUERY SELECT names, results;
END;
$$ LANGUAGE plpgsql;
-- If the variable of interest is just a rate return it as such,
-- otherwise normalize it to the census block area and return that
CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetPoints(
geom geometry,
geom_table_name text,
data_table_info cdb_observatory.OBS_ColumnData[]
)
RETURNS NUMERIC[]
AS $$
DECLARE
result NUMERIC[];
query text;
i int;
geoid text;
area NUMERIC;
BEGIN
-- TODO: does 'geoid' need to be generalized to geom_ref??
EXECUTE
format('SELECT geoid
FROM observatory.%I
WHERE ST_WITHIN($1, the_geom)',
geom_table_name)
USING geom
INTO geoid;
RAISE NOTICE 'geoid is %, geometry table is % ', geoid, geom_table_name;
EXECUTE
format('SELECT ST_Area(the_geom::geography) / (1000 * 1000)
FROM observatory.%I
WHERE geoid = %L',
geom_table_name,
geoid)
INTO area;
IF area IS NULL
THEN
RAISE NOTICE 'No geometry at %', ST_AsText(geom);
END IF;
query := 'SELECT Array[';
FOR i IN 1..array_upper(data_table_info, 1)
LOOP
IF area is NULL OR area = 0
THEN
-- give back null values
query := query || format('NULL::numeric ');
ELSIF ((data_table_info)[i]).aggregate != 'sum'
THEN
-- give back full variable
query := query || format('%I ', ((data_table_info)[i]).colname);
ELSE
-- give back variable normalized by area of geography
query := query || format('%I/%s ',
((data_table_info)[i]).colname,
area);
END IF;
IF i < array_upper(data_table_info, 1)
THEN
query := query || ',';
END IF;
END LOOP;
query := query || format(' ]::numeric[]
FROM observatory.%I
WHERE %I.geoid = %L
',
((data_table_info)[1]).tablename,
((data_table_info)[1]).tablename,
geoid
);
EXECUTE
query
INTO result
USING geom;
RETURN result;
END;
$$ LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetPolygons(
geom geometry,
geom_table_name text,
data_table_info cdb_observatory.OBS_ColumnData[]
)
RETURNS NUMERIC[]
AS $$
DECLARE
result NUMERIC[];
q_select text;
q_sum text;
q text;
i NUMERIC;
BEGIN
q_select := 'SELECT geoid, ';
q_sum := 'SELECT Array[';
FOR i IN 1..array_upper(data_table_info, 1)
LOOP
q_select := q_select || format( '%I ', ((data_table_info)[i]).colname);
IF ((data_table_info)[i]).aggregate ='sum'
THEN
q_sum := q_sum || format('sum(overlap_fraction * COALESCE(%I, 0)) ',((data_table_info)[i]).colname,((data_table_info)[i]).colname);
ELSE
q_sum := q_sum || ' NULL::numeric ';
END IF;
IF i < array_upper(data_table_info,1)
THEN
q_select := q_select || format(',');
q_sum := q_sum || format(',');
END IF;
END LOOP;
q = format('
WITH _overlaps As (
SELECT ST_Area(
ST_Intersection($1, a.the_geom)
) / ST_Area(a.the_geom) As overlap_fraction,
geoid
FROM observatory.%I As a
WHERE $1 && a.the_geom
),
values As (
', geom_table_name);
q := q || q_select || format('FROM observatory.%I ', ((data_table_info)[1].tablename));
q := q || ' ) ' || q_sum || ' ]::numeric[] FROM _overlaps, values
WHERE values.geoid = _overlaps.geoid';
EXECUTE
q
INTO result
USING geom;
RETURN result;
END;
$$ LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION OBS_GetSegmentSnapshot(geom geometry, geometry_level text default '"us.census.tiger".census_tract')
RETURNS json
AS $$
BEGIN
RETURN row_to_json(cdb_observatory._OBS_GetSegmentSnapshot(geom, geometry_level));
END;
$$ LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION _OBS_GetSegmentSnapshot(
geom geometry,
geometry_level text DEFAULT '"us.census.tiger".census_tract'
)
RETURNS TABLE(
segment_name TEXT,
total_pop_quantile NUMERIC,
male_pop_quantile NUMERIC,
female_pop_quantile NUMERIC,
median_age_quantile NUMERIC,
white_pop_quantile NUMERIC,
black_pop_quantile NUMERIC,
asian_pop_quantile NUMERIC,
hispanic_pop_quantile NUMERIC,
not_us_citizen_pop_quantile NUMERIC,
workers_16_and_over_quantile NUMERIC,
commuters_by_car_truck_van_quantile NUMERIC,
commuters_by_public_transportation_quantile NUMERIC,
commuters_by_bus_quantile NUMERIC,
commuters_by_subway_or_elevated_quantile NUMERIC,
walked_to_work_quantile NUMERIC,
worked_at_home_quantile NUMERIC,
children_quantile NUMERIC,
households_quantile NUMERIC,
population_3_years_over_quantile NUMERIC,
in_school_quantile NUMERIC,
in_grades_1_to_4_quantile NUMERIC,
in_grades_5_to_8_quantile NUMERIC,
in_grades_9_to_12_quantile NUMERIC,
in_undergrad_college_quantile NUMERIC,
pop_25_years_over_quantile NUMERIC,
high_school_diploma_quantile NUMERIC,
bachelors_degree_quantile NUMERIC,
masters_degree_quantile NUMERIC,
pop_5_years_over_quantile NUMERIC,
speak_only_english_at_home_quantile NUMERIC,
speak_spanish_at_home_quantile NUMERIC,
pop_determined_poverty_status_quantile NUMERIC,
poverty_quantile NUMERIC,
median_income_quantile NUMERIC,
gini_index_quantile NUMERIC,
income_per_capita_quantile NUMERIC,
housing_units_quantile NUMERIC,
vacant_housing_units_quantile NUMERIC,
vacant_housing_units_for_rent_quantile NUMERIC,
vacant_housing_units_for_sale_quantile NUMERIC,
median_rent_quantile NUMERIC,
percent_income_spent_on_rent_quantile NUMERIC,
owner_occupied_housing_units_quantile NUMERIC,
million_dollar_housing_units_quantile NUMERIC
)
AS $$
DECLARE
target_cols text[];
seg_name Text;
geom_id Text;
q Text;
BEGIN
target_cols := Array[
'"us.census.acs".B01001001_quantile',
'"us.census.acs".B01001002_quantile',
'"us.census.acs".B01001026_quantile',
'"us.census.acs".B01002001_quantile',
'"us.census.acs".B03002003_quantile',
'"us.census.acs".B03002004_quantile',
'"us.census.acs".B03002006_quantile',
'"us.census.acs".B03002012_quantile',
'"us.census.acs".B05001006_quantile',--
'"us.census.acs".B08006001_quantile',--
'"us.census.acs".B08006002_quantile',--
'"us.census.acs".B08006008_quantile',--
'"us.census.acs".B08006009_quantile',--
'"us.census.acs".B08006011_quantile',--
'"us.census.acs".B08006015_quantile',--
'"us.census.acs".B08006017_quantile',--
'"us.census.acs".B09001001_quantile',--
'"us.census.acs".B11001001_quantile',
'"us.census.acs".B14001001_quantile',--
'"us.census.acs".B14001002_quantile',--
'"us.census.acs".B14001005_quantile',--
'"us.census.acs".B14001006_quantile',--
'"us.census.acs".B14001007_quantile',--
'"us.census.acs".B14001008_quantile',--
'"us.census.acs".B15003001_quantile',
'"us.census.acs".B15003017_quantile',
'"us.census.acs".B15003022_quantile',
'"us.census.acs".B15003023_quantile',
'"us.census.acs".B16001001_quantile',--
'"us.census.acs".B16001002_quantile',--
'"us.census.acs".B16001003_quantile',--
'"us.census.acs".B17001001_quantile',--
'"us.census.acs".B17001002_quantile',--
'"us.census.acs".B19013001_quantile',
'"us.census.acs".B19083001_quantile',
'"us.census.acs".B19301001_quantile',
'"us.census.acs".B25001001_quantile',
'"us.census.acs".B25002003_quantile',
'"us.census.acs".B25004002_quantile',
'"us.census.acs".B25004004_quantile',
'"us.census.acs".B25058001_quantile',
'"us.census.acs".B25071001_quantile',
'"us.census.acs".B25075001_quantile',
'"us.census.acs".B25075025_quantile'
];
EXECUTE
$query$
SELECT (categories)[1]
FROM cdb_observatory._OBS_GetCategories(
$1,
Array['"us.census.spielman_singleton_segments".X10'],
$2)
LIMIT 1
$query$
INTO segment_name
USING geom, geometry_level;
q :=
format($query$
WITH a As (
SELECT
names As names,
vals As vals
FROM cdb_observatory._OBS_Get($1,
$2,
'2009 - 2013',
$3)
), percentiles As (
%s
FROM a)
SELECT $4, percentiles.*
FROM percentiles
$query$, cdb_observatory._OBS_BuildSnapshotQuery(target_cols));
RETURN QUERY
EXECUTE
q
USING geom, target_cols, geometry_level, segment_name;
END;
$$ LANGUAGE plpgsql;
--Get categorical variables from point
CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetCategories(
geom geometry,
dimension_names text[],
geometry_level text DEFAULT '"us.census.tiger".block_group',
time_span text DEFAULT '2009 - 2013'
)
RETURNS TABLE(names text[], categories text[]) as $$
DECLARE
geom_table_name text;
geoid text;
names text[];
results text[];
query text;
data_table_info cdb_observatory.OBS_ColumnData[];
BEGIN
geom_table_name := cdb_observatory._OBS_GeomTable(geom, geometry_level);
IF geom_table_name IS NULL
THEN
RAISE NOTICE 'Point % is outside of the data region', ST_AsText(geom);
RETURN QUERY SELECT '{}'::text[], '{}'::text[];
END IF;
data_table_info := cdb_observatory._OBS_GetColumnData(geometry_level,
dimension_names,
time_span);
names := (SELECT array_agg((d).colname)
FROM unnest(data_table_info) As d);
EXECUTE
format('SELECT geoid
FROM observatory.%I
WHERE the_geom && $1',
geom_table_name)
USING geom
INTO geoid;
query := 'SELECT ARRAY[';
FOR i IN 1..array_upper(data_table_info, 1)
LOOP
query = query || format('%I ', lower(((data_table_info)[i]).colname));
IF i < array_upper(data_table_info, 1)
THEN
query := query || ',';
END IF;
END LOOP;
query := query || format(' ]::text[]
FROM observatory.%I
WHERE %I.geoid = %L
',
((data_table_info)[1]).tablename,
((data_table_info)[1]).tablename,
geoid
);
EXECUTE
query
INTO results
USING geom;
RETURN QUERY
SELECT names,results
RETURN;
END;
$$ LANGUAGE plpgsql;
-- Placeholder for permission tweaks at creation time.
-- Make sure by default there are no permissions for publicuser
-- NOTE: this happens at extension creation time, as part of an implicit transaction.
-- REVOKE ALL PRIVILEGES ON SCHEMA cdb_observatory FROM PUBLIC, publicuser CASCADE;
-- Grant permissions on the schema to publicuser (but just the schema)
-- GRANT USAGE ON SCHEMA cdb_crankshaft TO publicuser;
-- Revoke execute permissions on all functions in the schema by default
-- REVOKE EXECUTE ON ALL FUNCTIONS IN SCHEMA cdb_observatory FROM PUBLIC, publicuser;

1723
release/observatory--0.0.2.sql Normal file
View File

File diff suppressed because it is too large Load Diff

5
release/observatory.control Normal file
View File

@@ -0,0 +1,5 @@
comment = 'CartoDB Observatory backend extension'
default_version = '0.0.2'
requires = 'postgis'
superuser = true
schema = cdb_observatory

19
scripts/README.md Normal file
View File

@@ -0,0 +1,19 @@
## Automatic tests and utilities
### Installation
Python 2.7 should cover you. Virtualenv recommended.
virtualenv venv
source venv/bin/activate
pip install -r requirements.txt
### Execution
Currently, we don't have direct access to the metadata end-to-end. This only
affects the generation of tests. As a stopgap, we have to define a connection
to the test Observatory account.
Run automated tests against a hostname:
(venv) OBS_HOSTNAME=<hostname.cartodb.com> OBS_API_KEY=<api_key> OBS_META_HOSTNAME=observatory.cartodb.com OBS_META_API_KEY= nosetests scripts/autotest.py

100
scripts/autotest.py Normal file
View File

@@ -0,0 +1,100 @@
from nose.tools import assert_equal, assert_is_not_none
from nose_parameterized import parameterized
import os
import re
import requests
HOSTNAME = os.environ['OBS_HOSTNAME']
API_KEY = os.environ['OBS_API_KEY']
META_HOSTNAME = os.environ.get('OBS_META_HOSTNAME', HOSTNAME)
META_API_KEY = os.environ.get('OBS_META_API_KEY', API_KEY)
USE_SCHEMA = 'OBS_USE_SCHEMA' in os.environ
def query(q, is_meta=False, **options):
'''
Query the account. Returned is the response, wrapped by the requests
library.
'''
url = 'https://{hostname}/api/v2/sql'.format(
hostname=META_HOSTNAME if is_meta else HOSTNAME)
params = options.copy()
params['q'] = re.sub(r'\s+', ' ', q)
params['api_key'] = META_API_KEY if is_meta else API_KEY
return requests.get(url, params=params)
MEASURE_COLUMNS = [(r['id'], ) for r in query('''
SELECT id FROM obs_column
WHERE type ILIKE 'numeric'
AND weight > 0
''', is_meta=True).json()['rows']]
CATEGORY_COLUMNS = [(r['id'], ) for r in query('''
SELECT id FROM obs_column
WHERE type ILIKE 'text'
AND weight > 0
''', is_meta=True).json()['rows']]
BOUNDARY_COLUMNS = [(r['id'], ) for r in query('''
SELECT id FROM obs_column
WHERE type ILIKE 'geometry'
AND weight > 0
''', is_meta=True).json()['rows']]
def default_point(column_id):
'''
Returns default test point for the column_id.
'''
if column_id == 'whosonfirst.wof_disputed_geom':
return 'CDB_LatLng(33.78, 76.57)'
elif column_id == 'whosonfirst.wof_marinearea_geom':
return 'CDB_LatLng(43.33, -68.47)'
elif column_id in ('us.census.tiger.school_district_elementary',
'us.census.tiger.school_district_secondary',
'us.census.tiger.school_district_elementary_clipped',
'us.census.tiger.school_district_secondary_clipped'):
return 'CDB_LatLng(40.7025, -73.7067)'
elif column_id.startswith('es.ine'):
return 'CDB_LatLng(40.39, -3.7)'
elif column_id.startswith('us.zillow'):
return 'CDB_LatLng(28.3305906291771, -81.3544048197256)'
else:
return 'CDB_LatLng(40.7, -73.9)'
@parameterized(MEASURE_COLUMNS)
def test_measure_points(column_id):
resp = query('''
SELECT * FROM {schema}OBS_GetMeasure({point}, '{column_id}')
'''.format(column_id=column_id,
schema='cdb_observatory.' if USE_SCHEMA else '',
point=default_point(column_id)))
assert_equal(resp.status_code, 200)
rows = resp.json()['rows']
assert_equal(1, len(rows))
assert_is_not_none(rows[0].values()[0])
@parameterized(CATEGORY_COLUMNS)
def test_category_points(column_id):
resp = query('''
SELECT * FROM {schema}OBS_GetCategory({point}, '{column_id}')
'''.format(column_id=column_id,
schema='cdb_observatory.' if USE_SCHEMA else '',
point=default_point(column_id)))
assert_equal(resp.status_code, 200)
rows = resp.json()['rows']
assert_equal(1, len(rows))
assert_is_not_none(rows[0].values()[0])
@parameterized(BOUNDARY_COLUMNS)
def test_boundary_points(column_id):
resp = query('''
SELECT * FROM {schema}OBS_GetBoundary({point}, '{column_id}')
'''.format(column_id=column_id,
schema='cdb_observatory.' if USE_SCHEMA else '',
point=default_point(column_id)))
assert_equal(resp.status_code, 200)
rows = resp.json()['rows']
assert_equal(1, len(rows))
assert_is_not_none(rows[0].values()[0])

25
scripts/generate_fixtures.py
View File

@@ -40,13 +40,15 @@ fixtures = [
('us.census.tiger.block_group', 'us.census.tiger.block_group', '2014'),
('us.census.tiger.zcta5', 'us.census.tiger.zcta5', '2014'),
('us.census.tiger.county', 'us.census.tiger.county', '2014'),
('us.census.acs.B01001001', 'us.census.tiger.census_tract', '2009 - 2013'),
('us.census.acs.B01001001_quantile', 'us.census.tiger.census_tract', '2009 - 2013'),
('us.census.acs.B01001001', 'us.census.tiger.block_group', '2009 - 2013'),
('us.census.acs.B01001001', 'us.census.tiger.block_group', '2010 - 2014'),
('us.census.spielman_singleton_segments.X10', 'us.census.tiger.census_tract', '2009 - 2013'),
('us.census.acs.B01003001', 'us.census.tiger.census_tract', '2010 - 2014'),
('us.census.acs.B01003001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
('us.census.acs.B01003001', 'us.census.tiger.block_group', '2010 - 2014'),
('us.census.spielman_singleton_segments.X10', 'us.census.tiger.census_tract', '2010 - 2014'),
('us.zillow.AllHomes_Zhvi', 'us.census.tiger.zcta5', '2014-01'),
('us.zillow.AllHomes_Zhvi', 'us.census.tiger.zcta5', '2016-03'),
('whosonfirst.wof_country_geom', 'whosonfirst.wof_country_geom', '2016'),
('us.census.tiger.zcta5_clipped', 'us.census.tiger.zcta5_clipped', '2014'),
('us.census.tiger.block_group_clipped', 'us.census.tiger.block_group_clipped', '2014'),
]
unique_tables = set()
@@ -75,10 +77,15 @@ with open('src/pg/test/fixtures/load_fixtures.sql', 'w') as outfile:
for tablename, colname, boundary_id in unique_tables:
if 'zcta5' in boundary_id:
where = '11%'
where = '\'11%\''
compare = 'LIKE'
elif 'whosonfirst' in boundary_id:
where = '(\'85632785\',\'85633051\',\'85633111\',\'85633147\',\'85633253\',\'85633267\')'
compare = 'IN'
else:
where = '36047%'
print ' '.join([select_star(tablename), "WHERE {} LIKE '{}'".format(colname, where)])
cdb.dump(' '.join([select_star(tablename), "WHERE {} LIKE '{}'".format(colname, where)]),
where = '\'36047%\''
compare = 'LIKE'
print ' '.join([select_star(tablename), "WHERE {}::text {} {}".format(colname, compare, where)])
cdb.dump(' '.join([select_star(tablename), "WHERE {}::text {} {}".format(colname, compare, where)]),
tablename, outfile, schema='observatory')
dropfiles.write('DROP TABLE IF EXISTS observatory.{};\n'.format(tablename))

3
scripts/requirements.txt Normal file
View File

@@ -0,0 +1,3 @@
requests
nose
nose_parameterized

72
src/pg/sql/41_observatory_augmentation.sql
View File

@@ -39,7 +39,7 @@ AS $$
boundary_id = 'us.census.tiger.block_group';
END IF;
target_cols := Array['us.census.acs.B01001001',
target_cols := Array['us.census.acs.B01003001',
'us.census.acs.B01001002',
'us.census.acs.B01001026',
'us.census.acs.B01002001',
@@ -171,7 +171,8 @@ BEGIN
THEN
RAISE NOTICE 'Point % is outside of the data region', ST_AsText(geom);
-- TODO this should return JSON
RETURN QUERY SELECT '{}'::text[], '{}'::NUMERIC[];
RETURN QUERY SELECT '{}'::json;
RETURN;
END IF;
IF data_table_info IS NULL THEN
@@ -198,6 +199,7 @@ BEGIN
SELECT unnest($1)
$query$
USING results;
RETURN;
END;
$$ LANGUAGE plpgsql;
@@ -398,7 +400,7 @@ BEGIN
IF time_span IS NULL THEN
-- TODO we should determine latest timespan for this measure
time_span := '2009 - 2013';
time_span := '2010 - 2014';
END IF;
EXECUTE '
@@ -492,7 +494,7 @@ DECLARE
result NUMERIC;
BEGIN
-- TODO use a super-column for global pop
population_measure_id := 'us.census.acs.B01001001';
population_measure_id := 'us.census.acs.B01003001';
EXECUTE format('SELECT cdb_observatory.OBS_GetMeasure(
%L, %L, %L, %L, %L
@@ -518,9 +520,37 @@ DECLARE
q_sum text;
q text;
i NUMERIC;
data_geoid_colname text;
geom_geoid_colname text;
BEGIN
q_select := 'SELECT geoid, ';
-- TODO we're assuming our geom_table has only one geom_ref column
-- we *really* should pass in both geom_table_name and boundary_id
-- TODO tablename should not be passed here (use boundary_id)
EXECUTE
format('SELECT ct.colname
FROM observatory.obs_column_to_column c2c,
observatory.obs_column_table ct,
observatory.obs_table t
WHERE c2c.reltype = ''geom_ref''
AND ct.column_id = c2c.source_id
AND ct.table_id = t.id
AND t.tablename = %L'
, (data_table_info)[1]->>'tablename')
INTO data_geoid_colname;
EXECUTE
format('SELECT ct.colname
FROM observatory.obs_column_to_column c2c,
observatory.obs_column_table ct,
observatory.obs_table t
WHERE c2c.reltype = ''geom_ref''
AND ct.column_id = c2c.source_id
AND ct.table_id = t.id
AND t.tablename = %L'
, geom_table_name)
INTO geom_geoid_colname;
q_select := format('SELECT %I, ', data_geoid_colname);
q_sum := 'SELECT Array[';
FOR i IN 1..array_upper(data_table_info, 1)
@@ -541,22 +571,22 @@ BEGIN
END IF;
END LOOP;
q = format('
q := format('
WITH _overlaps As (
SELECT ST_Area(
ST_Intersection($1, a.the_geom)
) / ST_Area(a.the_geom) As overlap_fraction,
geoid
%I
FROM observatory.%I As a
WHERE $1 && a.the_geom
),
values As (
', geom_table_name);
', geom_geoid_colname, geom_table_name);
q := q || q_select || format('FROM observatory.%I ', ((data_table_info)[1]->>'tablename'));
q := q || ' ) ' || q_sum || ' ]::numeric[] FROM _overlaps, values
WHERE values.geoid = _overlaps.geoid';
q := format(q || ' ) ' || q_sum || ' ]::numeric[] FROM _overlaps, values
WHERE values.%I = _overlaps.%I', geom_geoid_colname, geom_geoid_colname);
EXECUTE
q
@@ -596,13 +626,13 @@ DECLARE
seg_name Text;
geom_id Text;
q Text;
segment_name Text;
segment_names Text[];
BEGIN
IF boundary_id IS NULL THEN
boundary_id = 'us.census.tiger.census_tract';
END IF;
target_cols := Array[
'us.census.acs.B01001001_quantile',
'us.census.acs.B01003001_quantile',
'us.census.acs.B01001002_quantile',
'us.census.acs.B01001026_quantile',
'us.census.acs.B01002001_quantile',
@@ -650,14 +680,13 @@ target_cols := Array[
EXECUTE
$query$
SELECT (_OBS_GetCategories)->>'name'
SELECT array_agg(_OBS_GetCategories->>'category')
FROM cdb_observatory._OBS_GetCategories(
$1,
Array['us.census.spielman_singleton_segments.X10'],
Array['us.census.spielman_singleton_segments.X10', 'us.census.spielman_singleton_segments.X55'],
$2)
LIMIT 1
$query$
INTO segment_name
INTO segment_names
USING geom, boundary_id;
q :=
@@ -668,14 +697,14 @@ target_cols := Array[
array_agg(_OBS_GET->>'value') As vals
FROM cdb_observatory._OBS_Get($1,
$2,
'2009 - 2013',
'2010 - 2014',
$3)
), percentiles As (
%s
FROM a)
SELECT row_to_json(r) FROM
( SELECT $4 as segment_name, percentiles.*
( SELECT $4 as x10_segment, $5 as x55_segment, percentiles.*
FROM percentiles) r
$query$, cdb_observatory._OBS_BuildSnapshotQuery(target_cols)) results;
@@ -683,7 +712,7 @@ target_cols := Array[
EXECUTE
q
into result
USING geom, target_cols, boundary_id, segment_name;
USING geom, target_cols, boundary_id, segment_names[1], segment_names[2];
return result;
@@ -709,7 +738,7 @@ DECLARE
BEGIN
IF time_span IS NULL THEN
time_span = '2009 - 2013';
time_span = '2010 - 2014';
END IF;
IF boundary_id IS NULL THEN
@@ -722,6 +751,7 @@ BEGIN
THEN
RAISE NOTICE 'Point % is outside of the data region', ST_AsText(geom);
RETURN QUERY SELECT '{}'::text[], '{}'::text[];
RETURN;
END IF;
EXECUTE '
@@ -735,6 +765,7 @@ BEGIN
THEN
RAISE NOTICE 'No data table found for this location';
RETURN QUERY SELECT NULL::json;
RETURN;
END IF;
EXECUTE
@@ -749,6 +780,7 @@ BEGIN
THEN
RAISE NOTICE 'No geometry id for this location';
RETURN QUERY SELECT NULL::json;
RETURN;
END IF;
query := 'SELECT ARRAY[';

20
src/pg/sql/44_observatory_geometries.sql
View File

@@ -161,9 +161,9 @@ BEGIN
RAISE NOTICE 'target_table: %, geoid_colname: %', target_table, geoid_colname;
-- return name of geometry id column
-- return geometry id column value
EXECUTE format(
'SELECT %I
'SELECT %I::text
FROM observatory.%I
WHERE ST_Intersects($1, the_geom)
LIMIT 1', geoid_colname, target_table)
@@ -274,6 +274,7 @@ BEGIN
THEN
RAISE NOTICE 'No boundaries found for bounding box ''%'' in ''%''', ST_AsText(geom), boundary_id;
RETURN QUERY SELECT NULL::geometry, NULL::text;
RETURN;
END IF;
RAISE NOTICE 'target_table: %', target_table;
@@ -281,11 +282,12 @@ BEGIN
-- return first boundary in intersections
RETURN QUERY
EXECUTE format(
'SELECT %I, %I
'SELECT %I, %I::text
FROM observatory.%I
WHERE ST_%s($1, the_geom)
', geom_colname, geoid_colname, target_table, overlap_type)
USING geom;
RETURN;
END;
$$ LANGUAGE plpgsql;
@@ -328,6 +330,7 @@ BEGIN
time_span,
overlap_type
);
RETURN;
END;
$$ LANGUAGE plpgsql;
@@ -380,6 +383,7 @@ BEGIN
circle_boundary,
boundary_id,
time_span);
RETURN;
END;
$$ LANGUAGE plpgsql;
@@ -405,7 +409,7 @@ BEGIN
RAISE EXCEPTION 'Overlap type ''%'' is not an accepted type (choose intersects, within, or contains)', overlap_type;
ELSIF ST_GeometryType(geom) NOT IN ('ST_Polygon', 'ST_MultiPolygon')
THEN
RAISE EXCEPTION 'Invalid geometry type (%), expecting ''ST_MultiPolygon'' or ''ST_Polygon''', ST_GeometryType(geom);
RAISE EXCEPTION 'Invalid geometry type (%), expecting ''ST_MultiPolygon'' or ''ST_Polygon''', ST_GeometryType(geom);
END IF;
SELECT * INTO geoid_colname, target_table, geom_colname
@@ -416,6 +420,7 @@ BEGIN
THEN
RAISE NOTICE 'No boundaries found for bounding box ''%'' in ''%''', ST_AsText(geom), boundary_id;
RETURN QUERY SELECT NULL::geometry, NULL::text;
RETURN;
END IF;
RAISE NOTICE 'target_table: %', target_table;
@@ -423,11 +428,12 @@ BEGIN
-- return first boundary in intersections
RETURN QUERY
EXECUTE format(
'SELECT ST_PointOnSurface(%I) As %s, %I
'SELECT ST_PointOnSurface(%I) As %s, %I::text
FROM observatory.%I
WHERE ST_%s($1, the_geom)
', geom_colname, geom_colname, geoid_colname, target_table, overlap_type)
USING geom;
RETURN;
END;
$$ LANGUAGE plpgsql;
@@ -469,6 +475,7 @@ BEGIN
boundary_id,
time_span,
overlap_type);
RETURN;
END;
$$ LANGUAGE plpgsql;
@@ -522,6 +529,7 @@ BEGIN
boundary_id,
time_span,
overlap_type);
RETURN;
END;
$$ LANGUAGE plpgsql;
@@ -557,9 +565,11 @@ BEGIN
geom_c.type ILIKE 'geometry' AND
geom_c.id = '%s'
$string$, boundary_id, boundary_id);
RETURN;
-- AND geom_t.timespan = '%s' <-- put in requested year
-- TODO: filter by clipped vs. not so appropriate tablename are unique
-- so the limit 1 can be removed
RETURN;
END;
$$ LANGUAGE plpgsql;

24
src/pg/test/expected/40_observatory_utility_test.out
View File

@@ -3,11 +3,11 @@
\i test/fixtures/load_fixtures.sql
SET client_min_messages TO WARNING;
\set ECHO none
_obs_geomtable
obs_fc050f0b8673cfe3c6aa1040f749eb40975691b7
_obs_geomtable_with_returned_table
t
(1 row)
_obs_geomtable
_obs_geomtable_with_null_response
t
(1 row)
test_get_obs_column_with_geoid_and_census_1|test_get_obs_column_with_geoid_and_census_2
t|t
@@ -15,15 +15,15 @@ t|t
obs_getcolumndata_missing_measure
t
(1 row)
_obs_buildsnapshotquery
SELECT vals[1] As total_pop, vals[2] As male_pop, vals[3] As female_pop, vals[4] As median_age
_obs_buildsnapshotquery_test_1
t
(1 row)
_obs_buildsnapshotquery
SELECT vals[1] As mandarin_orange
_obs_buildsnapshotquery_test_2
t
(1 row)
_obs_getrelatedcolumn
{es.ine.total_pop,NULL,us.census.acs.B01001001}
_obs_getrelatedcolumn_test
t
(1 row)
_obs_standardizemeasurename
test_343_2_qqq
_obs_standardizemeasurename_test
t
(1 row)

41
src/pg/test/expected/42_observatory_exploration_test.out
View File

@@ -2,17 +2,36 @@
SET client_min_messages TO WARNING;
\set ECHO none
_obs_searchtables_tables_match|_obs_searchtables_timespan_matches
obs_1babf5a26a1ecda5fb74963e88408f71d0364b81|t
t|t
(1 row)
_obs_searchtables_timespan_does_not_match
t
(1 row)
obs_search
(es.ine.total_pop,"The total number of all people living in a geographic area.","Total Population",sum,)
(us.census.acs.B01001001,"The total number of all people living in a given geographic area. This is a very useful catch-all denominator when calculating rates.","Total Population",sum,)
(us.census.acs.B01001001_quantile,"The total number of all people living in a given geographic area. This is a very useful catch-all denominator when calculating rates.","Quantile:Total Population",quantile,)
(3 rows)
id|description|name|aggregate|source
us.census.acs.B01003001_quantile|The total number of all people living in a given geographic area. This is a very useful catch-all denominator when calculating rates.|Quantile:Total Population|quantile|
es.ine.total_pop|The total number of all people living in a geographic area.|Total Population|sum|
us.census.acs.B01003001|The total number of all people living in a given geographic area. This is a very useful catch-all denominator when calculating rates.|Total Population|sum|
us.census.acs.B19301001|Per capita income is the mean income computed for every man, woman, and child in a particular group. It is derived by dividing the total income of a particular group by the total population.|Per Capita Income in the past 12 Months|average|
us.census.acs.B01001001_quantile|The total number of all people living in a given geographic area. This is a very useful catch-all denominator when calculating rates.|Quantile:Total Population|quantile|
(5 rows)
boundary_id|description|time_span|tablename
us.census.tiger.zcta5|ZCTAs are approximate area representations of U.S. Postal Service (USPS) five-digit ZIP Code service areas that the Census Bureau creates using whole blocks to present statistical data from censuses and surveys. The Census Bureau defines ZCTAs by allocating each block that contains addresses to a single ZCTA, usually to the ZCTA that reflects the most frequently occurring ZIP Code for the addresses within that tabulation block. Blocks that do not contain addresses but are completely surrounded by a single ZCTA (enclaves) are assigned to the surrounding ZCTA; those surrounded by multiple ZCTAs will be added to a single ZCTA based on limited buffering performed between multiple ZCTAs. The Census Bureau identifies five-digit ZCTAs using a five-character numeric code that represents the most frequently occurring USPS ZIP Code within that ZCTA, and this code may contain leading zeros.
There are significant changes to the 2010 ZCTA delineation from that used in 2000. Coverage was extended to include the Island Areas for 2010 so that the United States, Puerto Rico, and the Island Areas have ZCTAs. Unlike 2000, when areas that could not be assigned to a ZCTA were given a generic code ending in “XX” (land area) or “HH” (water area), for 2010 there is no universal coverage by ZCTAs, and only legitimate five-digit areas are defined. The 2010 ZCTAs will better represent the actual Zip Code service areas because the Census Bureau initiated a process before creation of 2010 blocks to add block boundaries that split polygons with large numbers of addresses using different Zip Codes.
Data users should not use ZCTAs to identify the official USPS ZIP Code for mail delivery. The USPS makes periodic changes to ZIP Codes to support more efficient mail delivery. The ZCTAs process used primarily residential addresses and was biased towards Zip Codes used for city-style mail delivery, thus there may be Zip Codes that are primarily nonresidential or boxes only that may not have a corresponding ZCTA.|2014|obs_144e8b4f906885b2e057ac4842644a553ae49c6e
whosonfirst.wof_continent_geom|Continents of the world.|2016|obs_9880042f935aab0d0e4b71fb6963d7726e64c534
whosonfirst.wof_country_geom| |2016|obs_1ea93bbc109c87c676b3270789dacf7a1430db6c
whosonfirst.wof_region_geom| |2016|obs_4fca4f060854cc3ae8c109999635a71bbde6964e
whosonfirst.wof_marinearea_geom| |2016|obs_5105018d57c69b8a7e064fc17a9485647b311a99
whosonfirst.wof_disputed_geom| |2016|obs_7dba9374b15fbab0c7bd7dca6dec6c4792fe86a3
us.census.tiger.congressional_district|Congressional districts are the 435 areas from which people are elected to the U.S. House of Representatives. After the apportionment of congressional seats among the states based on decennial census population counts, each state with multiple seats is responsible for establishing congressional districts for the purpose of electing representatives. Each congressional district is to be as equal in population to all other congressional districts in a state as practicable. For the District of Columbia, Puerto Rico, and each Island Area, a separate code is used to identify the entire areas of these state-equivalent entities as having a single nonvoting delegate.|2014|obs_01b608b3a4ca503ad7acc0b1f84817bc1da3e193
us.census.tiger.congressional_district_clipped|A cartography-ready version of US Congressional Districts|2014|obs_9d258128ff4288eb9a6d7e5c0adbef8ef0172a86
us.census.tiger.school_district_unified_clipped|A cartography-ready version of Unified School District|2014|obs_546285f2c636f5380e7bfbb0c0db67863f6ed41d
us.census.tiger.cbsa_clipped|A cartography-ready version of Core Based Statistical Area (CBSA)|2014|obs_3512a78ca8c7e9b5fbd5390c4ed1638b9938fcbf
us.census.tiger.block|Census blocks are statistical areas bounded by visible features, such as streets, roads, streams, and railroad tracks, and by nonvisible boundaries, such as selected property lines and city, township, school district, and county limits and short line-of-sight extensions of streets and roads. Generally, census blocks are small in area; for example, a block in a city bounded on all sides by streets. Census blocks in suburban and rural areas may be large, irregular, and bounded by a variety of features, such as roads, streams, and transmission lines. In remote areas, census blocks may encompass hundreds of square miles. Census blocks cover the entire territory of the United States, Puerto Rico, and the Island Areas. Census blocks nest within all other tabulated census geographic entities and are the basis for all tabulated data.|2014|obs_ffebc3eb689edab4faa757f75ca02c65d7db7327
us.census.tiger.school_district_elementary_clipped|A cartography-ready version of Elementary School District|2014|obs_41d99a86857c05f63320ea44419a68831e74ac3d
us.census.tiger.block_group_clipped|A cartography-ready version of US Census Block Groups|2014|obs_6c1309a64d8f3e6986061f4d1ca7b57743e75e74
us.census.tiger.census_tract_clipped|A cartography-ready version of US Census Tracts|2014|obs_fcd4e4f5610f6764973ef8c0c215b2e80bec8963
us.census.tiger.zcta5_clipped|A cartography-ready version of US Census Zip Code Tabulation Areas|2014|obs_7615e8622a68bfc5fe37c69c9880edfb40250103
us.census.tiger.place|Incorporated places are those reported to the Census Bureau as legally in existence as of January 1, 2010, as reported in the latest Boundary and Annexation Survey (BAS), under the laws of their respective states. An incorporated place is established to provide governmental functions for a concentration of people as opposed to a minor civil division, which generally is created to provide services or administer an area without regard, necessarily, to population. Places always are within a single state or equivalent entity, but may extend across county and county subdivision boundaries. An incorporated place usually is a city, town, village, or borough, but can have other legal descriptions. For Census Bureau data tabulation and presentation purposes, incorporated places exclude:
Boroughs in Alaska (treated as statistical equivalents of counties).
Towns in the New England states, New York, and Wisconsin (treated as MCDs).
@@ -20,9 +39,13 @@ Boroughs in New York (treated as MCDs).
Census Designated Places (CDPs) are the statistical counterparts of incorporated places, and are delineated to provide data for settled concentrations of population that are identifiable by name but are not legally incorporated under the laws of the state in which they are located. The boundaries usually are defined in cooperation with local or tribal officials and generally updated prior to each decennial census. These boundaries, which usually coincide with visible features or the boundary of an adjacent incorporated place or another legal entity boundary, have no legal status, nor do these places have officials elected to serve traditional municipal functions. CDP boundaries may change from one decennial census
to the next with changes in the settlement pattern; a CDP with the same name as in an earlier census does not necessarily have the same boundary. CDPs must be contained within a single state and may not extend into an incorporated place. There are no population size requirements for CDPs.
Hawaii is the only state that has no incorporated places recognized by the Census Bureau. All places shown in decennial census data products for Hawaii are CDPs. By agreement with the state of Hawaii, the Census Bureau does not show data separately for the city of Honolulu, which is coextensive with Honolulu County. In Puerto Rico, which also does not have incorporated places, the Census Bureau recognizes only CDPs and refers to them as comunidades or zonas urbanas. Guam also has only CDPs.|2014|obs_76a52df2018de8d064f1a99f93544473927cb7ae
us.census.tiger.place_clipped|A cartography-ready version of Incorporated Places|2014|obs_db91d46d317a4ffcf509efca8e5e3a42d29e0792
us.census.tiger.school_district_secondary_clipped|A cartography-ready version of Secondary School District|2014|obs_63b15ba742ccea136ddb88427cc37fee218702d4
us.census.tiger.cbsa|Core Based Statistical Areas (CBSAs) consist of the county or counties or equivalent entities associated with at least one core (urbanized area or urban cluster) of at least 10,000 population, plus adjacent counties having a high degree of social and economic integration with the core as measured through commuting ties with the counties associated with the core. The general concept of a CBSA is that of a
core area containing a substantial population nucleus, together with adjacent communities having a high degree of economic and social integration with that core. The term “core based statistical area” became effective in 2003 and refers collectively to metropolitan statistical areas and micropolitan statistical areas. The U.S. Office of Management and Budget (OMB) defines CBSAs to provide a nationally consistent set of geographic entities for the United States and Puerto Rico for use in tabulating and presenting statistical data. Current CBSAs are based on application of the 2000 standards (published in the Federal Register of December 27, 2000) with Census 2000 data. The first set of areas defined based on the 2000 standards were announced on June 6, 2003; subsequent updates have been made to the universe of CBSAs and related statistical areas. No CBSAs are defined in the Island Areas. Statistical areas related to CBSAs include metropolitan divisions, combined statistical areas (CSAs), New England city and town areas (NECTAs), NECTA divisions, and combined NECTAs.|2014|obs_c75be9ef45e87c789c3607dd9aeef6094d5e5109
us.census.tiger.puma_clipped|A cartography-ready version of US Census Public Use Microdata Areas|2014|obs_dc244bf520f62e4a09e290a02e55368fd0758f95
us.census.tiger.county|The primary legal divisions of most states are termed counties. In Louisiana, these divisions are known as parishes. In Alaska, which has no counties, the equivalent entities are the organized boroughs, city and boroughs, municipalities, and census areas; the latter of which are delineated cooperatively for statistical purposes by the state of Alaska and the Census Bureau. In four states (Maryland, Missouri, Nevada, and Virginia), there are one or more incorporated places that are independent of any county organization and thus constitute primary divisions of their states. These incorporated places are known as independent cities and are treated as equivalent entities for purposes of data presentation. The District of Columbia and Guam have no primary divisions, and each area is considered an equivalent entity for purposes of data presentation. All of the counties in Connecticut and Rhode Island and nine counties in Massachusetts were dissolved as functioning governmental entities; however, the Census Bureau continues to present data for these historical entities in order to provide comparable geographic units at the county level of the geographic hierarchy for these states and represents them as nonfunctioning legal entities in data products. The Census Bureau treats the following entities as equivalents of counties for purposes of data presentation: municipios in Puerto Rico, districts and islands in American Samoa, municipalities in the Commonwealth of the Northern Mariana Islands, and islands in the U.S. Virgin Islands. Each county or statistically equivalent entity is assigned a three-character numeric Federal Information Processing Series (FIPS) code based on alphabetical sequence that is unique within state and an eight-digit National Standard feature identifier.|2014|obs_1babf5a26a1ecda5fb74963e88408f71d0364b81
us.census.tiger.county_clipped|A cartography-ready version of US County|2014|obs_23cb5063486bd7cf36f17e89e5e65cd31b331f6e
us.census.tiger.school_district_unified|School Districts are geographic entities within which state, county, local officials, the Bureau of Indian Affairs, or the U.S. Department of Defense provide public educational services for the area’s residents. The Census Bureau obtains the boundaries, names, local education agency codes, and school district levels for school districts from state and local school officials for the primary purpose of providing the U.S. Department of Education with estimates of the number of children “at risk” within each school district, county, and state. This information serves as the basis for the Department of Education to determine the annual allocation of Title I funding to states and school districts.
The Census Bureau tabulates data for three types of school districts: elementary, secondary, and unified. Each school district is assigned a five-digit code that is unique within state. School district codes are the local education agency number assigned by the Department of Education and are not necessarily in alphabetical order by school district name.
The elementary school districts provide education to the lower grade/age levels and the secondary school districts provide education to the upper grade/age levels. Unified school districts provide education to children of all school ages in their service areas. In general, where there is a unified school district, no elementary or secondary school district exists; and where there is an elementary school district, the secondary school district may or may not exist.
@@ -41,9 +64,7 @@ The Census Bureau’s representation of school districts in various data product
us.census.tiger.block_group|Block groups (BGs) are statistical divisions of census tracts, are generally defined to contain between 600 and 3,000 people, and are used to present data and control block numbering. A block group consists of clusters of blocks within the same census tract that have the same first digit of their four-digit census block number. For example, blocks 3001, 3002, 3003, ..., 3999 in census tract 1210.02 belong to BG 3 in that census tract. Most BGs were delineated by local participants in the Census Bureau’s Participant Statistical Areas Program. The Census Bureau delineated BGs only where a local or tribal government declined to participate, and a regional organization or State Data Center was not available to participate.
A BG usually covers a contiguous area. Each census tract contains at least one BG, and BGs are uniquely numbered within the census tract. Within the standard census geographic hierarchy, BGs never cross state, county, or census tract boundaries but may cross the boundaries of any other geographic entity. Tribal census tracts and tribal BGs are separate and unique geographic areas defined within federally recognized American Indian reservations and can cross state and county boundaries (see “Tribal Census Tract” and “Tribal Block Group”). The tribal census tracts and tribal block groups may be completely different from the census tracts and block groups defined by state and county.|2014|obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308
us.census.tiger.state|States and Equivalent Entities are the primary governmental divisions of the United States. In addition to the 50 states, the Census Bureau treats the District of Columbia, Puerto Rico, American Samoa, the Commonwealth of the Northern Mariana Islands, Guam, and the U.S. Virgin Islands as the statistical equivalents of states for the purpose of data presentation.|2014|obs_624e5d2362e08aaa5463d7671e7748432262719c
us.census.tiger.zcta5|ZCTAs are approximate area representations of U.S. Postal Service (USPS) five-digit ZIP Code service areas that the Census Bureau creates using whole blocks to present statistical data from censuses and surveys. The Census Bureau defines ZCTAs by allocating each block that contains addresses to a single ZCTA, usually to the ZCTA that reflects the most frequently occurring ZIP Code for the addresses within that tabulation block. Blocks that do not contain addresses but are completely surrounded by a single ZCTA (enclaves) are assigned to the surrounding ZCTA; those surrounded by multiple ZCTAs will be added to a single ZCTA based on limited buffering performed between multiple ZCTAs. The Census Bureau identifies five-digit ZCTAs using a five-character numeric code that represents the most frequently occurring USPS ZIP Code within that ZCTA, and this code may contain leading zeros.
There are significant changes to the 2010 ZCTA delineation from that used in 2000. Coverage was extended to include the Island Areas for 2010 so that the United States, Puerto Rico, and the Island Areas have ZCTAs. Unlike 2000, when areas that could not be assigned to a ZCTA were given a generic code ending in “XX” (land area) or “HH” (water area), for 2010 there is no universal coverage by ZCTAs, and only legitimate five-digit areas are defined. The 2010 ZCTAs will better represent the actual Zip Code service areas because the Census Bureau initiated a process before creation of 2010 blocks to add block boundaries that split polygons with large numbers of addresses using different Zip Codes.
Data users should not use ZCTAs to identify the official USPS ZIP Code for mail delivery. The USPS makes periodic changes to ZIP Codes to support more efficient mail delivery. The ZCTAs process used primarily residential addresses and was biased towards Zip Codes used for city-style mail delivery, thus there may be Zip Codes that are primarily nonresidential or boxes only that may not have a corresponding ZCTA.|2014|obs_144e8b4f906885b2e057ac4842644a553ae49c6e
us.census.tiger.state_clipped|A cartography-ready version of US States|2014|obs_f39f1d7cd5a22b87140860cbd58539f1591a1810
us.census.tiger.puma|PUMAs are geographic areas for which the Census Bureau provides selected extracts of raw data from a small sample of census records that are screened to protect confidentiality. These extracts are referred to as public use microdata sample (PUMS) files.
For the 2010 Census, each state, the District of Columbia, Puerto Rico, and some Island Area participants delineated PUMAs for use in presenting PUMS data based on a 5 percent sample of decennial census or American Community Survey data. These areas are required to contain at least 100,000 people. This is different from Census 2000 when two types of PUMAs were defined: a 5 percent PUMA as for 2010 and an additional super-PUMA designed to provide a 1 percent sample. The PUMAs are identified by a five-digit census code unique within state.|2014|obs_7c9493c41fa8f4bd178ab993ea3d5891c1977667
(11 rows)
(30 rows)

3
src/pg/test/expected/44_observatory_geometries_test.out
View File

@@ -57,6 +57,9 @@ t
obs_getboundariesbygeometry_tracts_around_null_island
t
(1 row)
obs_getboundariesbygeometry_wof
t
(1 row)
obs_getboundariesbypointandradius_around_cartodb
t
(1 row)

10
src/pg/test/fixtures/drop_fixtures.sql vendored
View File

@@ -7,13 +7,15 @@ DROP TABLE IF EXISTS observatory.obs_column_tag;
DROP TABLE IF EXISTS observatory.obs_tag;
DROP TABLE IF EXISTS observatory.obs_column_to_column;
DROP TABLE IF EXISTS observatory.obs_65f29658e096ca1485bf683f65fdbc9f05ec3c5d;
DROP TABLE IF EXISTS observatory.obs_1746e37b7cd28cb131971ea4187d42d71f09c5f3;
DROP TABLE IF EXISTS observatory.obs_1a098da56badf5f32e336002b0a81708c40d29cd;
DROP TABLE IF EXISTS observatory.obs_fc050f0b8673cfe3c6aa1040f749eb40975691b7;
DROP TABLE IF EXISTS observatory.obs_7615e8622a68bfc5fe37c69c9880edfb40250103;
DROP TABLE IF EXISTS observatory.obs_1babf5a26a1ecda5fb74963e88408f71d0364b81;
DROP TABLE IF EXISTS observatory.obs_8764a6b439a4f8714f54d4b3a157bc5e36519066;
DROP TABLE IF EXISTS observatory.obs_3e7cc9cfd403b912c57b42d5f9195af9ce2f3cdb;
DROP TABLE IF EXISTS observatory.obs_d34555209878e8c4b37cf0b2b3d072ff129ec470;
DROP TABLE IF EXISTS observatory.obs_b393b5b88c6adda634b2071a8005b03c551b609a;
DROP TABLE IF EXISTS observatory.obs_1ea93bbc109c87c676b3270789dacf7a1430db6c;
DROP TABLE IF EXISTS observatory.obs_fc050f0b8673cfe3c6aa1040f749eb40975691b7;
DROP TABLE IF EXISTS observatory.obs_6c1309a64d8f3e6986061f4d1ca7b57743e75e74;
DROP TABLE IF EXISTS observatory.obs_d39f7fe5959891c8296490d83c22ded31c54af13;
DROP TABLE IF EXISTS observatory.obs_144e8b4f906885b2e057ac4842644a553ae49c6e;
DROP TABLE IF EXISTS observatory.obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308;
DROP TABLE IF EXISTS observatory.obs_ab038198aaab3f3cb055758638ee4de28ad70146;

29926
src/pg/test/fixtures/load_fixtures.sql vendored
View File

File diff suppressed because one or more lines are too long

31
src/pg/test/sql/40_observatory_utility_test.sql
View File

@@ -10,7 +10,7 @@ SELECT
ST_SetSRID(ST_Point(-74.0059, 40.7128), 4326),
'us.census.tiger.census_tract',
'2014'
);
) = 'obs_fc050f0b8673cfe3c6aa1040f749eb40975691b7' As _obs_geomtable_with_returned_table;
-- get null for unknown geometry_id
-- should give back null
@@ -18,7 +18,7 @@ SELECT
cdb_observatory._OBS_GeomTable(
ST_SetSRID(ST_Point(-74.0059, 40.7128), 4326),
'us.census.tiger.nonexistant_id' -- not in catalog
);
) IS NULL _obs_geomtable_with_null_response;
-- future test: give back nulls when geometry doesn't intersect
-- SELECT
@@ -27,21 +27,16 @@ SELECT
-- 'us.census.tiger.census_tract'
-- );
-- OBS_GetColumnData
-- should give back:
-- colname | tablename | aggregate
-- -----------|-----------------|-----------
-- geoid | obs_{hex table} | null
-- total_pop | obs_{hex table} | sum
WITH result as (
SELECT
array_agg(a) expected from cdb_observatory._OBS_GetColumnData(
'us.census.tiger.census_tract',
Array['us.census.tiger.census_tract_geoid', 'us.census.acs.B01001001'],
'2009 - 2013') a
Array['us.census.spielman_singleton_segments.X55', 'us.census.acs.B01003001'],
'2010 - 2014') a
)
select (expected)[1]::text = '{"colname":"geoid","tablename":"obs_d34555209878e8c4b37cf0b2b3d072ff129ec470","aggregate":null,"name":"US Census Tracts Geoids","type":"Text","description":null,"boundary_id":"us.census.tiger.census_tract"}' as test_get_obs_column_with_geoid_and_census_1,
(expected)[2]::text = '{"colname":"geoid","tablename":"obs_ab038198aaab3f3cb055758638ee4de28ad70146","aggregate":null,"name":"US Census Tracts Geoids","type":"Text","description":null,"boundary_id":"us.census.tiger.census_tract"}' as test_get_obs_column_with_geoid_and_census_2
select
(expected)[1]::text = '{"colname":"x55","tablename":"obs_65f29658e096ca1485bf683f65fdbc9f05ec3c5d","aggregate":null,"name":"Spielman-Singleton Segments: 55 Clusters","type":"Text","description":"Sociodemographic classes from Spielman and Singleton 2015, 55 clusters","boundary_id":"us.census.tiger.census_tract"}' as test_get_obs_column_with_geoid_and_census_1,
(expected)[2]::text = '{"colname":"total_pop","tablename":"obs_b393b5b88c6adda634b2071a8005b03c551b609a","aggregate":"sum","name":"Total Population","type":"Numeric","description":"The total number of all people living in a given geographic area. This is a very useful catch-all denominator when calculating rates.","boundary_id":"us.census.tiger.census_tract"}' as test_get_obs_column_with_geoid_and_census_2
from result;
-- should be null-valued
@@ -50,7 +45,7 @@ SELECT
array_agg(a) expected from cdb_observatory._OBS_GetColumnData(
'us.census.tiger.census_tract',
Array['us.census.tiger.baloney'],
'2009 - 2013') a
'2010 - 2014') a
)
select expected is null as OBS_GetColumnData_missing_measure
from result;
@@ -60,24 +55,24 @@ from result;
SELECT
cdb_observatory._OBS_BuildSnapshotQuery(
Array['total_pop','male_pop','female_pop','median_age']
);
) = 'SELECT vals[1] As total_pop, vals[2] As male_pop, vals[3] As female_pop, vals[4] As median_age' As _OBS_BuildSnapshotQuery_test_1;
-- should give back: SELECT vals[1] As mandarin_orange
SELECT
cdb_observatory._OBS_BuildSnapshotQuery(
Array['mandarin_orange']
);
) = 'SELECT vals[1] As mandarin_orange' As _OBS_BuildSnapshotQuery_test_2;
SELECT cdb_observatory._OBS_GetRelatedColumn(
Array[
'es.ine.pop_0_4',
'us.census.acs.B01001001',
'us.census.acs.B01003001',
'us.census.acs.B01001002'
],
'denominator'
);
) = '{es.ine.total_pop,NULL,us.census.acs.B01003001}' As _OBS_GetRelatedColumn_test;
-- should give back a standardized measure name
SELECT cdb_observatory._OBS_StandardizeMeasureName('test 343 %% 2 qqq }}{{}}');
SELECT cdb_observatory._OBS_StandardizeMeasureName('test 343 %% 2 qqq }}{{}}') = 'test_343_2_qqq' As _OBS_StandardizeMeasureName_test;
\i test/fixtures/drop_fixtures.sql

23
src/pg/test/sql/41_observatory_augmentation_test.sql
View File

@@ -6,14 +6,9 @@
WITH result as(
Select count(coalesce(OBS_GetDemographicSnapshot->>'value', 'foo')) expected_columns
FROM cdb_observatory.OBS_GetDemographicSnapshot(cdb_observatory._TestPoint())
) select expected_columns ='59' as OBS_GetDemographicSnapshot_test_no_returns
) select expected_columns = 52 as OBS_GetDemographicSnapshot_test_no_returns
FROM result;
--
-- names | vals
-- --------------|-------
-- median_income | 45122
WITH result as (
SELECT _OBS_Get::text as expected FROM
cdb_observatory._OBS_Get(
@@ -103,13 +98,13 @@ SELECT cdb_observatory.OBS_GetSegmentSnapshot(
cdb_observatory._TestPoint(),
'us.census.tiger.census_tract'
)::text =
'{"segment_name":"Spielman-Singleton Segments: 10 Clusters","us.census.acs.B01001001_quantile":"0.234783783783784","us.census.acs.B01001002_quantile":"0.422405405405405","us.census.acs.B01001026_quantile":"0.0987567567567568","us.census.acs.B01002001_quantile":"0.0715","us.census.acs.B03002003_quantile":"0.295310810810811","us.census.acs.B03002004_quantile":"0.407189189189189","us.census.acs.B03002006_quantile":"0.625608108108108","us.census.acs.B03002012_quantile":"0.795202702702703","us.census.acs.B05001006_quantile":"0.703797297297297","us.census.acs.B08006001_quantile":"0.59227027027027","us.census.acs.B08006002_quantile":"0.0180540540540541","us.census.acs.B08006008_quantile":"0.993756756756757","us.census.acs.B08006009_quantile":"0.728162162162162","us.census.acs.B08006011_quantile":"0.995972972972973","us.census.acs.B08006015_quantile":"0.929135135135135","us.census.acs.B08006017_quantile":"0.625432432432432","us.census.acs.B09001001_quantile":"0.0386081081081081","us.census.acs.B11001001_quantile":"0.157121621621622","us.census.acs.B14001001_quantile":"0.241878378378378","us.census.acs.B14001002_quantile":"0.173783783783784","us.census.acs.B14001005_quantile":"0.0380675675675676","us.census.acs.B14001006_quantile":"0.0308108108108108","us.census.acs.B14001007_quantile":"0.0486216216216216","us.census.acs.B14001008_quantile":"0.479743243243243","us.census.acs.B15003001_quantile":"0.297675675675676","us.census.acs.B15003017_quantile":"0.190351351351351","us.census.acs.B15003022_quantile":"0.802513513513514","us.census.acs.B15003023_quantile":"0.757148648648649","us.census.acs.B16001001_quantile":"0.255405405405405","us.census.acs.B16001002_quantile":"0.196094594594595","us.census.acs.B16001003_quantile":"0.816851351351351","us.census.acs.B17001001_quantile":"0.252513513513514","us.census.acs.B17001002_quantile":"0.560054054054054","us.census.acs.B19013001_quantile":"0.777472972972973","us.census.acs.B19083001_quantile":"0.336932432432432","us.census.acs.B19301001_quantile":"0.655378378378378","us.census.acs.B25001001_quantile":"0.141810810810811","us.census.acs.B25002003_quantile":"0.362824324324324","us.census.acs.B25004002_quantile":"0.463837837837838","us.census.acs.B25004004_quantile":"0","us.census.acs.B25058001_quantile":"0.939040540540541","us.census.acs.B25071001_quantile":"0.419445945945946","us.census.acs.B25075001_quantile":"0.0387972972972973","us.census.acs.B25075025_quantile":"0"}' as test_point_segmentation;
'{"x10_segment":"Wealthy, urban without Kids","x55_segment":"Wealthy transplants displacing long-term local residents","us.census.acs.B01003001_quantile":"0.3235","us.census.acs.B01001002_quantile":"0.494716216216216","us.census.acs.B01001026_quantile":"0.183756756756757","us.census.acs.B01002001_quantile":"0.0752837837837838","us.census.acs.B03002003_quantile":"0.293162162162162","us.census.acs.B03002004_quantile":"0.455527027027027","us.census.acs.B03002006_quantile":"0.656405405405405","us.census.acs.B03002012_quantile":"0.840081081081081","us.census.acs.B05001006_quantile":"0.727135135135135","us.census.acs.B08006001_quantile":"0.688635135135135","us.census.acs.B08006002_quantile":"0.0204459459459459","us.census.acs.B08006008_quantile":"0.679324324324324","us.census.acs.B08006009_quantile":"0.996716216216216","us.census.acs.B08006011_quantile":"0.967418918918919","us.census.acs.B08006015_quantile":"0.512945945945946","us.census.acs.B08006017_quantile":"0.0504864864864865","us.census.acs.B09001001_quantile":"0.192405405405405","us.census.acs.B11001001_quantile":"0.331702702702703","us.census.acs.B14001001_quantile":"0.296283783783784","us.census.acs.B14001002_quantile":"0.045472972972973","us.census.acs.B14001005_quantile":"0.0442702702702703","us.census.acs.B14001006_quantile":"0.0829054054054054","us.census.acs.B14001007_quantile":"0.701135135135135","us.census.acs.B14001008_quantile":"0.404527027027027","us.census.acs.B15003001_quantile":"0.191824324324324","us.census.acs.B15003017_quantile":"0.864162162162162","us.census.acs.B15003022_quantile":"0.754297297297297","us.census.acs.B15003023_quantile":"0.350054054054054","us.census.acs.B16001001_quantile":"0.217635135135135","us.census.acs.B16001002_quantile":"0.85972972972973","us.census.acs.B16001003_quantile":"0.342851351351351","us.census.acs.B17001001_quantile":"0.51204054054054","us.census.acs.B17001002_quantile":"0.813540540540541","us.census.acs.B19013001_quantile":"0.0948648648648649","us.census.acs.B19083001_quantile":"0.678351351351351","us.census.acs.B19301001_quantile":"0.146108108108108","us.census.acs.B25001001_quantile":"0.149067567567568","us.census.acs.B25002003_quantile":"0","us.census.acs.B25004002_quantile":"0","us.census.acs.B25004004_quantile":"0.944554054054054","us.census.acs.B25058001_quantile":"0.398040540540541","us.census.acs.B25071001_quantile":"0.0596081081081081","us.census.acs.B25075001_quantile":"0","us.census.acs.B25075025_quantile":null}' as test_point_segmentation;
-- segmentation around null island
SELECT cdb_observatory.OBS_GetSegmentSnapshot(
ST_SetSRID(ST_Point(0, 0), 4326),
'us.census.tiger.census_tract'
)::text = '{"segment_name":null,"us.census.acs.B01001001_quantile":null,"us.census.acs.B01001002_quantile":null,"us.census.acs.B01001026_quantile":null,"us.census.acs.B01002001_quantile":null,"us.census.acs.B03002003_quantile":null,"us.census.acs.B03002004_quantile":null,"us.census.acs.B03002006_quantile":null,"us.census.acs.B03002012_quantile":null,"us.census.acs.B05001006_quantile":null,"us.census.acs.B08006001_quantile":null,"us.census.acs.B08006002_quantile":null,"us.census.acs.B08006008_quantile":null,"us.census.acs.B08006009_quantile":null,"us.census.acs.B08006011_quantile":null,"us.census.acs.B08006015_quantile":null,"us.census.acs.B08006017_quantile":null,"us.census.acs.B09001001_quantile":null,"us.census.acs.B11001001_quantile":null,"us.census.acs.B14001001_quantile":null,"us.census.acs.B14001002_quantile":null,"us.census.acs.B14001005_quantile":null,"us.census.acs.B14001006_quantile":null,"us.census.acs.B14001007_quantile":null,"us.census.acs.B14001008_quantile":null,"us.census.acs.B15003001_quantile":null,"us.census.acs.B15003017_quantile":null,"us.census.acs.B15003022_quantile":null,"us.census.acs.B15003023_quantile":null,"us.census.acs.B16001001_quantile":null,"us.census.acs.B16001002_quantile":null,"us.census.acs.B16001003_quantile":null,"us.census.acs.B17001001_quantile":null,"us.census.acs.B17001002_quantile":null,"us.census.acs.B19013001_quantile":null,"us.census.acs.B19083001_quantile":null,"us.census.acs.B19301001_quantile":null,"us.census.acs.B25001001_quantile":null,"us.census.acs.B25002003_quantile":null,"us.census.acs.B25004002_quantile":null,"us.census.acs.B25004004_quantile":null,"us.census.acs.B25058001_quantile":null,"us.census.acs.B25071001_quantile":null,"us.census.acs.B25075001_quantile":null,"us.census.acs.B25075025_quantile":null}' as null_island_segmentation;
)::text = '{"x10_segment":null,"x55_segment":null,"us.census.acs.B01003001_quantile":null,"us.census.acs.B01001002_quantile":null,"us.census.acs.B01001026_quantile":null,"us.census.acs.B01002001_quantile":null,"us.census.acs.B03002003_quantile":null,"us.census.acs.B03002004_quantile":null,"us.census.acs.B03002006_quantile":null,"us.census.acs.B03002012_quantile":null,"us.census.acs.B05001006_quantile":null,"us.census.acs.B08006001_quantile":null,"us.census.acs.B08006002_quantile":null,"us.census.acs.B08006008_quantile":null,"us.census.acs.B08006009_quantile":null,"us.census.acs.B08006011_quantile":null,"us.census.acs.B08006015_quantile":null,"us.census.acs.B08006017_quantile":null,"us.census.acs.B09001001_quantile":null,"us.census.acs.B11001001_quantile":null,"us.census.acs.B14001001_quantile":null,"us.census.acs.B14001002_quantile":null,"us.census.acs.B14001005_quantile":null,"us.census.acs.B14001006_quantile":null,"us.census.acs.B14001007_quantile":null,"us.census.acs.B14001008_quantile":null,"us.census.acs.B15003001_quantile":null,"us.census.acs.B15003017_quantile":null,"us.census.acs.B15003022_quantile":null,"us.census.acs.B15003023_quantile":null,"us.census.acs.B16001001_quantile":null,"us.census.acs.B16001002_quantile":null,"us.census.acs.B16001003_quantile":null,"us.census.acs.B17001001_quantile":null,"us.census.acs.B17001002_quantile":null,"us.census.acs.B19013001_quantile":null,"us.census.acs.B19083001_quantile":null,"us.census.acs.B19301001_quantile":null,"us.census.acs.B25001001_quantile":null,"us.census.acs.B25002003_quantile":null,"us.census.acs.B25004002_quantile":null,"us.census.acs.B25004004_quantile":null,"us.census.acs.B25058001_quantile":null,"us.census.acs.B25071001_quantile":null,"us.census.acs.B25075001_quantile":null,"us.census.acs.B25075025_quantile":null}' as null_island_segmentation;
WITH result as (
SELECT array_agg(_OBS_GetCategories) as expected FROM
@@ -150,15 +145,15 @@ SELECT abs(OBS_GetMeasure_zhvi_point_default_latest - 972900) / 972900 < 0.001 A
SELECT abs(OBS_GetMeasure_total_pop_point - 10923.093200390833950) / 10923.093200390833950 < 0.001 As OBS_GetMeasure_total_pop_point_test FROM
cdb_observatory.OBS_GetMeasure(
cdb_observatory._TestPoint(),
'us.census.acs.B01001001'
'us.census.acs.B01003001'
) As t(OBS_GetMeasure_total_pop_point);
-- Poly-based OBS_GetMeasure, default normalization (none)
-- is result within 0.1% of expected
SELECT abs(OBS_GetMeasure_total_pop_polygon - 12327.3133495107) / 12327.3133495107 < 0.001 As OBS_GetMeasure_total_pop_polygon_test FROM
SELECT abs(OBS_GetMeasure_total_pop_polygon - 9833.47316573952) / 9833.47316573952 < 0.001 As OBS_GetMeasure_total_pop_polygon_test FROM
cdb_observatory.OBS_GetMeasure(
cdb_observatory._TestArea(),
'us.census.acs.B01001001'
'us.census.acs.B01003001'
) As t(OBS_GetMeasure_total_pop_polygon);
-- Point-based OBS_GetMeasure with denominator normalization
@@ -169,7 +164,7 @@ SELECT (abs(cdb_observatory.OBS_GetMeasure(
-- Poly-based OBS_GetMeasure with denominator normalization
SELECT abs(cdb_observatory.OBS_GetMeasure(
cdb_observatory._TestArea(),
'us.census.acs.B01001002', 'denominator') - 0.49026340444793965457) / 0.49026340444793965457 < 0.001 As OBS_GetMeasure_total_male_poly_denominator;
'us.census.acs.B01001002', 'denominator') - 0.50597531462834994530) / 0.49026340444793965457 < 0.001 As OBS_GetMeasure_total_male_poly_denominator;
-- Point-based OBS_GetCategory
SELECT cdb_observatory.OBS_GetCategory(
@@ -186,7 +181,7 @@ SELECT (abs(OBS_GetPopulation - 10923.093200390833950) / 10923.093200390833950)
) As m(OBS_GetPopulation);
-- Poly-based OBS_GetPopulation, default normalization (none)
SELECT (abs(obs_getpopulation_polygon - 12327.3133495107) / 12327.3133495107) < 0.001 As obs_getpopulation_polygon_test
SELECT (abs(obs_getpopulation_polygon - 9833.47316573952) / 9833.47316573952) < 0.001 As obs_getpopulation_polygon_test
FROM
cdb_observatory.OBS_GetPopulation(
cdb_observatory._TestArea()
@@ -198,7 +193,7 @@ SELECT (abs(cdb_observatory.obs_getuscensusmeasure(
-- Poly-based OBS_GetUSCensusMeasure, default normalization (none)
SELECT (abs(cdb_observatory.obs_getuscensusmeasure(
cdb_observatory._testarea(), 'male population') - 6043.63061042765) / 6043.63061042765) < 0.001 As obs_getuscensusmeasure;
cdb_observatory._testarea(), 'male population') - 4975.49467892449) / 4975.49467892449) < 0.001 As obs_getuscensusmeasure;
-- Point-based OBS_GetUSCensusCategory
SELECT cdb_observatory.OBS_GetUSCensusCategory(

10
src/pg/test/sql/42_observatory_exploration_test.sql
View File

@@ -8,8 +8,8 @@
\set cartodb_county_geometry ''
-- _OBS_SearchTables tests
SELECT
t.table_name As _OBS_SearchTables_tables_match,
SELECT
t.table_name = 'obs_1babf5a26a1ecda5fb74963e88408f71d0364b81' As _OBS_SearchTables_tables_match,
t.timespan = '2014' As _OBS_SearchTables_timespan_matches
FROM cdb_observatory._OBS_SearchTables(
'us.census.tiger.county',
@@ -24,8 +24,10 @@ FROM cdb_observatory._OBS_SearchTables(
'1988' -- year before first tiger data was collected
) As t(table_name, timespan);
SELECT cdb_observatory.OBS_Search('total_pop');
SELECT *
FROM cdb_observatory.OBS_Search('total_pop');
SELECT * from cdb_observatory.OBS_GetAvailableBoundaries(cdb_observatory._TestPoint());
SELECT *
FROM cdb_observatory.OBS_GetAvailableBoundaries(cdb_observatory._TestPoint());
\i test/fixtures/drop_fixtures.sql

11
src/pg/test/sql/44_observatory_geometries_test.sql
View File

@@ -171,6 +171,17 @@ FROM (
ORDER BY geom_refs ASC
) As m(the_geom, geom_refs);
-- who's on first boundaries
SELECT
array_agg(geom_refs) = Array['85632785','85633051','85633111','85633147','85633253','85633267'] As OBS_GetBoundariesByGeometry_wof
FROM (
SELECT *
FROM cdb_observatory.OBS_GetBoundariesByGeometry(
ST_MakeEnvelope(-4.66, 40.43, 14.48, 51.99, 4326),
'whosonfirst.wof_country_geom')
ORDER BY geom_refs ASC
) As m(the_geom, geom_refs);
-- OBS_GetBoundariesByPointAndRadius
-- check that all census tracts intersecting with the geometry are returned