cartodb4/observatory-extension
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge branch 'develop' into release-v1-alpha

Browse Source
This commit is contained in:
Mario de Frutos
2016-05-18 08:44:01 +02:00
parent 9cddac7768 fe3a6759f7
commit 6a5e299fa5
8 changed files with 73 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
doc/API.md
View File

@@ -8,3 +8,4 @@ This file is for reference purposes only. It is intended for tracking the Data O
* [Boundary Functions](boundary_functions.md)
* [Discovery Functions](discovery_functions.md)
* [Glossary](glossary.md)
* [License](license.md)

25
doc/boundary_functions.md
View File

@@ -1,6 +1,8 @@
# Boundary Functions
Use the following functions to retrieve [Boundary](/cartodb-platform/dataobservatory/overview/#boundary-data) data. Data ranges from small areas (e.g. US Census Block Groups) to large areas (e.g. Countries). You can access boundaries by point location lookup, bounding box lookup, direct ID access and several other methods described below.
Use the following functions to retrieve [Boundary](/cartodb-platform/data/overview/#boundary-data) data. Data ranges from small areas (e.g. US Census Block Groups) to large areas (e.g. Countries). You can access boundaries by point location lookup, bounding box lookup, direct ID access and several other methods described below.
You can [access](/cartodb-platform/data/accessing/#accessing-the-data-observatory) boundaries through the CartoDB Editor. The same methods will work if you are using the CartoDB Platform to develop your application. We [encourage you](/cartodb-platform/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
## OBS_GetBoundariesByGeometry(polygon geometry, geometry_id text)
@@ -28,10 +30,10 @@ If geometries are not found for the requested `polygon`, `geometry_id`, `timespa
#### Example
Insert all Census Tracts from Lower Manhattan and nearby areas within the supplied bounding box to a table named `manhattan_census_tracts` which has columns `the_geom` (geometry) and `geoid` (text).
Insert all Census Tracts from Lower Manhattan and nearby areas within the supplied bounding box to a table named `manhattan_census_tracts` which has columns `the_geom` (geometry) and `geom_refs` (text).
```sql
INSERT INTO manhattan_census_tracts(the_geom, geoid)
INSERT INTO manhattan_census_tracts(the_geom, geom_refs)
SELECT *
FROM OBS_GetBoundariesByGeometry(
ST_MakeEnvelope(-74.0251922607,40.6945658517,
@@ -67,14 +69,14 @@ Column Name | Description
the_geom | a point geometry on a boundary (e.g., a point that lies on a US Census tract)
geom_refs| a string identifier for the geometry (e.g., the geoid of a US Census tract)
If geometries are not found for the requested geometry, `geometry_id`, `timespan`, or `overlap_type`, then null values are returned.
If geometries are not found for the requested geometry, `geometry_id`, `timespan`, or `overlap_type`, then NULL values are returned.
#### Example
Insert points that lie on Census Tracts from Lower Manhattan and nearby areas within the supplied bounding box to a table named `manhattan_census_tracts` which has columns `the_geom` (geometry) and `geoid` (text).
Insert points that lie on Census Tracts from Lower Manhattan and nearby areas within the supplied bounding box to a table named `manhattan_tract_points` which has columns `the_geom` (geometry) and `geom_refs` (text).
```sql
INSERT INTO manhattan_census_tract_points(the_geom, geoid)
INSERT INTO manhattan_tract_points (the_geom, geom_refs)
SELECT *
FROM OBS_GetPointsByGeometry(
ST_MakeEnvelope(-74.0251922607,40.6945658517,
@@ -175,9 +177,10 @@ geom | a WGS84 polygon geometry
#### Example
Use a table of `geometry_id`s (e.g., geoid from the U.S. Census) to select the unique boundaries that they correspond to.
Use a table of `geometry_id`s (e.g., geoid from the U.S. Census) to select the unique boundaries that they correspond to and insert into a table called, `overlapping_polygons`. This is a useful method for creating new choropleths of aggregate data.
```SQL
INSERT INTO overlapping_polygons (the_geom, geometry_id, point_count)
SELECT
OBS_GetBoundaryById(geometry_id, 'us.census.tiger.county') As the_geom,
geometry_id,
@@ -213,10 +216,10 @@ If geometries are not found for the requested point and radius, `geometry_id`, `
#### Example
Insert into table `denver_census_tracts` the census tract boundaries and geoids of census tracts which intersect within 10 miles of downtown Denver, Colorado.
Insert into table `denver_census_tracts` the census tract boundaries and geom_refs of census tracts which intersect within 10 miles of downtown Denver, Colorado.
```sql
INSERT INTO denver_census_tracts(the_geom, geoid)
INSERT INTO denver_census_tracts(the_geom, geom_refs)
SELECT *
FROM OBS_GetBoundariesByPointAndRadius(
CDB_LatLng(39.7392, -104.9903), -- Denver, Colorado
@@ -255,10 +258,10 @@ If geometries are not found for the requested point and radius, `geometry_id`, `
#### Example
Insert into table `denver_census_tracts` points on US census tracts and their corresponding geoids for census tracts which intersect within 10 miles of downtown Denver, Colorado, USA.
Insert into table `denver_tract_points` points on US census tracts and their corresponding geoids for census tracts which intersect within 10 miles of downtown Denver, Colorado, USA.
```sql
INSERT INTO denver_census_tracts(the_geom, geoid)
INSERT INTO denver_tract_points(the_geom, geom_refs)
SELECT *
FROM OBS_GetPointsByPointAndRadius(
CDB_LatLng(39.7392, -104.9903), -- Denver, Colorado

14
doc/discovery_functions.md
View File

@@ -1,6 +1,6 @@
# Discovery Functions
If you are using the [discovery methods](/cartodb-platform/dataobservatory/overview/#discovery-methods) from the Data Observatory, use the following functions to retrieve [boundary](/cartodb-platform/dataobservatory/overview/#boundary-data) and [measures](/cartodb-platform/dataobservatory/overview/#measures-data) data.
If you are using the [discovery methods](/cartodb-platform/data/overview/#discovery-methods) from the Data Observatory, use the following functions to retrieve [boundary](/cartodb-platform/data/overview/#boundary-data) and [measures](/cartodb-platform/data/overview/#measures-data) data.
## OBS_Search(search_term)
@@ -19,16 +19,16 @@ A TABLE containing the following properties
Key | Description
--- | ---
measure_id | the unique id of the measure for use with the ```OBS_GetMeasure``` function
id | the unique id of the measure for use with the ```OBS_GetMeasure``` function
name | the human readable name of the measure
description | a brief description of the measure
aggregate_type | **sum** are raw count values, **median** are statistical medians, **average** are statistical averages, **undefined** other (e.g. an index value)
sources | where the data came from (e.g. US Census Bureau)
aggregate | **sum** are raw count values, **median** are statistical medians, **average** are statistical averages, **undefined** other (e.g. an index value)
source | where the data came from (e.g. US Census Bureau)
#### Example
```SQL
SELECT * FROM OBS_Search('inequality')
SELECT * FROM OBS_Search('home value')
```
## OBS_GetAvailableBoundaries(point_geometry)
@@ -47,9 +47,9 @@ A TABLE containing the following properties
Key | Description
--- | ---
boundary_id | a boundary identifier from the [boundary ID glossary](/cartodb-platform/dataobservatory/glossary/#boundary-ids)
boundary_id | a boundary identifier from the [boundary ID glossary](/cartodb-platform/data/glossary/#boundary-ids)
description | a brief description of the boundary dataset
timespan | the timespan attached the boundary. this does not mean that the boundary is invalid outside of the timespan, but is the explicit timespan published with the geometry.
time_span | the timespan attached the boundary. this does not mean that the boundary is invalid outside of the timespan, but is the explicit timespan published with the geometry.
#### Example

16
doc/glossary.md
View File

@@ -1,6 +1,6 @@
# Glossary
A list of boundary ids and measure_names for Data Observatory functions. For US based boundaries, the Shoreline Clipped version provides a high-quality shoreline clipping for mapping uses.
A list of boundary ids and measure_names for Data Observatory functions. For US based boundaries, the Shoreline Clipped version provides a high-quality shoreline clipping for mapping uses.
## Boundary IDs
@@ -18,12 +18,12 @@ US Census Blocks | us.census.tiger.block | us.census.tiger.block_clipped
US Census Block Groups | us.census.tiger.block_group | us.census.tiger.block_group_clipped
US Census PUMAs | us.census.tiger.puma | us.census.tiger.puma_clipped
US Incorporated Places | us.census.tiger.place | us.census.tiger.place_clipped
ES Sección Censal | es.ine.geom |
Regions (First-level Administrative) | whosonfirst.wof_region_geom |
Continents | whosonfirst.wof_continent_geom |
Countries | whosonfirst.wof_country_geom |
Marine Areas | whosonfirst.wof_marinearea_geom |
Disputed Areas | whosonfirst.wof_disputed_geom |
ES Sección Censal | es.ine.geom | none
Regions (First-level Administrative) | whosonfirst.wof_region_geom | none
Continents | whosonfirst.wof_continent_geom | none
Countries | whosonfirst.wof_country_geom | none
Marine Areas | whosonfirst.wof_marinearea_geom | none
Disputed Areas | whosonfirst.wof_disputed_geom | none
@@ -67,7 +67,7 @@ Number of workers with a commute between 25 and 29 minutes | The number of w
Number of workers with a commute between 30 and 34 minutes | The number of workers over the age of 16 who do not work from home and commute in between 30 and 34 minutes in a geographic area.
Number of workers with a commute between 45 and 59 minutes | The number of workers over the age of 16 who do not work from home and commute in between 45 and 59 minutes in a geographic area.
Children under 18 Years of Age | The number of people within each geography who are under 18 years of age.
Households | A count of the number of households in each geography. A household consists of one or more people who live in the same dwelling and also share at meals or living accommodation, and may consist of a single family or some other grouping of people.
Households | A count of the number of households in each geography. A household consists of one or more people who live in the same dwelling and also share at meals or living accommodation, and may consist of a single family or some other grouping of people.
Population 15 Years and Over | The number of people in a geographic area who are over the age of 15. This is used mostly as a denominator of marital status.
Never Married | The number of people in a geographic area who have never been married.
Currently married | The number of people in a geographic area who are currently married.

18
doc/license.md Normal file
View File

@@ -0,0 +1,18 @@
# License
The Data Observatory is a collection of various sources of data with varying licenses. We have worked hard to find you data that will work for the broadest set of use-cases. For competency, please still review the terms for any dataset you use and respect the rights of the owners for each dataset. The following third-party data sources are used in the Data Observatory, and we have included the links to the terms governing their use.
Name | Terms link
-------|---------
ACS | [https://www.usa.gov/government-works](https://www.usa.gov/government-works)
TIGER | [https://www.usa.gov/government-works](https://www.usa.gov/government-works)
Zillow Home Value Index | [http://www.zillow.com/corp/Terms.htm](http://www.zillow.com/corp/Terms.htm)
Who's on First | [http://whosonfirst.mapzen.com#License](http://whosonfirst.mapzen.com#License)
GeoNames | [http://www.geonames.org/](http://www.geonames.org/)
GeoPlanet | [https://developer.yahoo.com/geo/geoplanet/](https://developer.yahoo.com/geo/geoplanet/)
Natural Earth | [http://www.naturalearthdata.com/about/terms-of-use/](http://www.naturalearthdata.com/about/terms-of-use/)
Quattroshapes | [https://github.com/foursquare/quattroshapes/blob/master/LICENSE.md](https://github.com/foursquare/quattroshapes/blob/master/LICENSE.md)
Zetashapes | [http://zetashapes.com/license](http://zetashapes.com/license)
Spielman & Singleton | [http://www.tandfonline.com/doi/full/10.1080/00045608.2015.1052335](http://www.tandfonline.com/doi/full/10.1080/00045608.2015.1052335)
Instituto Nacional de Estadistica | [http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout](http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout)

34
doc/measures_functions.md
View File

@@ -1,21 +1,21 @@
# Measures Functions
[Data Observatory Measures](/cartodb-platform/dataobservatory/overview/#measures-methods) are the numerical location data you can access. The measure Functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (Point or Polygon) and receiving back a measure (an absolute value) for that location.
[Data Observatory Measures](/cartodb-platform/data/overview/#measures-methods) are the numerical location data you can access. The measure functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (point or polygon) and receiving back a measure (an absolute value) for that location.
There are hundreds of Measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf).
There are hundreds of measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf).
We show here how you can access Measures through the CartoDB Editor. The same methods will work if you are using the CartoDB Platform to develop your application. We encourage you to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
You can [access](/cartodb-platform/data/accessing/#accessing-the-data-observatory) measures through the CartoDB Editor. The same methods will work if you are using the CartoDB Platform to develop your application. We [encourage you](/cartodb-platform/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
## OBS_GetUSCensusMeasure(point geometry, measure_name text);
## OBS_GetUSCensusMeasure(point geometry, measure_name text)
The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure based on a subset of the US Census variables at a point location. The ```OBS_GetUSCensusMeasure``` function is limited to only a subset of all Measures that are available in the Data Observatory, to access the full list, use the ```OBS_GetMeasure``` function below.
The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure based on a subset of the US Census variables at a point location. The ```OBS_GetUSCensusMeasure``` function is limited to only a subset of all measures that are available in the Data Observatory, to access the full list, use measure IDs with the ```OBS_GetMeasure``` function below.
#### Arguments
Name |Description
--- | ---
point | a WGS84 point geometry (the_geom)
measure_name | a human readable name of a US Census variable. The list of measure_names is [available in the glossary](/cartodb-platform/dataobservatory/glossary/#obsgetuscensusmeasure-names-table).
measure_name | a human readable name of a US Census variable. The list of measure_names is [available in the glossary](/cartodb-platform/data/glossary/#obsgetuscensusmeasure-names-table).
normalize | for measures that are are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://cartodb.github.io/bigmetadata/index.html) (optional)
boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
time_span | time span of interest (e.g., 2010 - 2014)
@@ -33,11 +33,11 @@ value | the raw or normalized measure
Add a measure to an empty numeric column based on point locations in your table.
```SQL
UPDATE tablename
SET local_male_population = OBS_GetUSCensusMeasure(the_geom, 'Male Population')
UPDATE tablename
SET total_population = OBS_GetUSCensusMeasure(the_geom, 'Total Population')
```
## OBS_GetUSCensusMeasure(polygon geometry, measure_name text);
## OBS_GetUSCensusMeasure(polygon geometry, measure_name text)
The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure based on a subset of the US Census variables within a given polygon. The ```OBS_GetUSCensusMeasure``` function is limited to only a subset of all measures that are available in the Data Observatory, to access the full list, use the ```OBS_GetUSCensusMeasure``` function below.
@@ -46,7 +46,7 @@ The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure
Name |Description
--- | ---
polygon | a WGS84 polygon geometry (the_geom)
measure_name | a human readable string name of a US Census variable. The list of measure_names is [available in the glossary](/cartodb-platform/dataobservatory/glossary/#obsgetuscensusmeasure-names-table).
measure_name | a human readable string name of a US Census variable. The list of measure_names is [available in the glossary](/cartodb-platform/data/glossary/#obsgetuscensusmeasure-names-table).
normalize | for measures that are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf) (optional)
boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
time_span | time span of interest (e.g., 2010 - 2014)
@@ -68,7 +68,7 @@ UPDATE tablename
SET local_male_population = OBS_GetUSCensusMeasure(the_geom, 'Male Population')
```
## OBS_GetMeasure(point geometry, measure_id text);
## OBS_GetMeasure(point geometry, measure_id text)
The ```OBS_GetMeasure(point, measure_id)``` function returns any Data Observatory measure at a point location. You can browse all available Measures in the [Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf)).
@@ -96,10 +96,10 @@ Add a measure to an empty numeric column based on point locations in your table
```SQL
UPDATE tablename
SET local_male_population = OBS_GetMeasure(the_geom, 'us.census.acs.B08134006')
SET median_home_value_sqft = OBS_GetMeasure(the_geom, 'us.zillow.AllHomes_MedianValuePerSqft')
```
## OBS_GetMeasure(polygon geometry, measure_id text);
## OBS_GetMeasure(polygon geometry, measure_id text)
The ```OBS_GetMeasure(polygon, measure_id)``` function returns any Data Observatory measure calculated within a polygon.
@@ -126,15 +126,15 @@ value | the raw or normalized measure
Add a measure to an empty column based on polygons in your table
```SQL
UPDATE tablename
SET local_male_population = OBS_GetMeasure(the_geom, 'us.census.acs.B08134006')
UPDATE tablename
SET household_count = OBS_GetMeasure(the_geom, 'us.census.acs.B11001001')
```
#### Errors
* If an unrecognized normalization type is input, raise an error: `'Only valid inputs for "normalize" are "area" (default) and "denominator".`
## OBS_GetCategory(point geometry, category_id text);
## OBS_GetCategory(point geometry, category_id text)
The ```OBS_GetCategory(point, category_id)``` function returns any Data Observatory Category value at a point location. The Categories available are currently limited to Segmentation categories. See the Segmentation section of the [Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf) for more detail.
@@ -158,6 +158,6 @@ value | a text based category found at the supplied point
Add the Category to an empty column text column based on point locations in your table
```SQL
UPDATE tablename
UPDATE tablename
SET segmentation = OBS_GetCategory(the_geom, 'us.census.spielman_singleton_segments.X55')
```

6
scripts/README.md
View File

@@ -10,6 +10,10 @@ Python 2.7 should cover you. Virtualenv recommended.
### Execution
Currently, we don't have direct access to the metadata end-to-end. This only
affects the generation of tests. As a stopgap, we have to define a connection
to the test Observatory account.
Run automated tests against a hostname:
(venv) OBS_HOSTNAME=<hostname.cartodb.com> OBS_API_KEY=foobar nosetests autotest.py
(venv) OBS_HOSTNAME=<hostname.cartodb.com> OBS_API_KEY=<api_key> OBS_META_HOSTNAME=observatory.cartodb.com OBS_META_API_KEY= nosetests scripts/autotest.py

6
scripts/autotest.py
View File

@@ -25,19 +25,19 @@ def query(q, is_meta=False, **options):
return requests.get(url, params=params)
MEASURE_COLUMNS = [(r['id'], ) for r in query('''
SELECT id FROM observatory.obs_column
SELECT id FROM obs_column
WHERE type ILIKE 'numeric'
AND weight > 0
''', is_meta=True).json()['rows']]
CATEGORY_COLUMNS = [(r['id'], ) for r in query('''
SELECT id FROM observatory.obs_column
SELECT id FROM obs_column
WHERE type ILIKE 'text'
AND weight > 0
''', is_meta=True).json()['rows']]
BOUNDARY_COLUMNS = [(r['id'], ) for r in query('''
SELECT id FROM observatory.obs_column
SELECT id FROM obs_column
WHERE type ILIKE 'geometry'
AND weight > 0
''', is_meta=True).json()['rows']]