How to publish and fix data-gov demos

From Data-gov Wiki

Jump to: navigation, search
Infobox (How-To) edit with form
  • name: How to publish and fix data-gov demos

  • description: step by step instructions for publishing data-gov demos. Technology bug fix information is also provided
  • creator(s): Li Ding
  • created: 2010/04/20
  • relation(s): How to build a data-gov demo
  • modified: 2010-5-17


Contents

Prerequisite

Principles
  • we are trying to build demos to educate web users and developers
  • our demo can be built in a quick and easy fashion
Your demo should answer the following questions
  • audience: Who cares about this demo?
  • motivation: Should normal web users care about this demo, and why?
  • highlights: Which part of this demo is interesting? Show a screen shot with explanations.
  • contributions: Is there any similar work? Which part of this demo is novel?
Please answer the following questions about the technologies used in the demo
  • technology: What technologies have been used and demonstrated in this demo? How it can be used to educate web developers?
  • replication: Can this demo be easily duplicated?
  • sweet spot: Which difficult problems have been overcome by the use of this technology? Explain.

Demo Page

demo page deployment

we suggest the following levels demo development

level url path notes
Level0: local version up to you
  • outside svn
  • you do local test first
  • try use relative URLs, or make absolute URLs configurable
Level1: testing version http://data-gov.tw.rpi.edu/demo/test/
  • outside svn
  • vi/scp
Level2: experimental version http://data-gov.tw.rpi.edu/demo/lab/
  • in svn
  • vi/scp or svn update
Level3: stable version http://data-gov.tw.rpi.edu/demo/stable/
  • in svn
  • svn update only. you need approval from Li or Jim to get your demo into that directory)

demo page filename

  • file name for main demo page should be constructed as the following
  demo-DATASETID-DEMO_SHORT_NAME.YOUR_FILE_EXTENSION
  e.g. http://data-gov.tw.rpi.edu/demo/stable/demo-1187-40x-wildfire-budget.html
  • file name for auxiliary demo page use the main filename (excluding file extension) as prefix
  demo-DATASETID-DEMO_SHORT_NAME-SUBNAME.sparql
  e.g. http://data-gov.tw.rpi.edu/demo/stable/demo-1187-40x-wildfire-budget-1187.sparql

demo page look and feel

copy the following CSS style sheet

  <!--  css style sheet -->
  <style type="text/css">
   img { border: 0; }
   .menuimg{ height:32px; width:32px; border:0px; position:relative; top:0px;}
   table.pretty {  border-collapse: separate;  border-spacing: 0.2em; }
   table.pretty th, table.pretty td { background: whitesmoke; border: 1px silver solid;  padding: 0.2em; }
   </style>

leverage the following demo header template, you need to make some changes

<!-- Demo Header -->		
<table width="990px">
	<tr>
	    <!-- title -->		
		<td align="center" valign="baseline"><h3>National Wildland Fire Statistics and Government Budget</h3></td>

		<td width="120" align="right">
	        <!-- link to data-gov home page -->		
			<a class="info" href="http://data-gov.tw.rpi.edu/">
			<img src="http://data-gov.tw.rpi.edu/images/logo-data-gov.png" alt="Data-gov" class="menuimg"></a>


	        <!-- link to demo's wiki page -->		
			<a class="info" href="http://data-gov.tw.rpi.edu/wiki/Demo:_Linking_Wildland_Fire_and_Government_Budget">
			<img src='http://data-gov.tw.rpi.edu/images/blue_info.png' class="menuimg" alt="wiki page"></a>
		</td>
	</tr>
</table>


leverage the following demo canvas template, you need to make some changes

    <div id='main_canvas' style="width: 1100px; height: 600px;">
      <p><img src="http://data-gov.tw.rpi.edu/images/ajax-loader.gif" alt="loading ..."></p>
    </div>

leverage the following demo description template, you need to make some changes

<!-- Demo Description -->		
<table width="1100px" class="pretty" >
<tr>
	<td>description</td>

	<td>
		This demo displays the number of fires and burned area (in acres) between 1960 and 2008. 
		It is connected to the sume of budget accounts on "Wildland Fire" from Department of the Interior (DOI) and Depart of Agriculture (USDA). 
		Meanwhile, it also loads external information from e.g. dbpedia.org and the Data-gov wiki. 
	</td>
</tr>

<tr>
	<td>creator</td>
	<td>
		<a href="http://data-gov.tw.rpi.edu/wiki/Li_Ding">Li Ding</a>
	</td>

</tr>

<tr>
	<td>created</td>
	<td>
	  2010/04/08 
	</td>
</tr>
	
<tr>
	<td>datasets</td>

	<td>
	 <div> * <a href="http://data-gov.tw.rpi.edu/wiki/Dataset_1187">Dataset 1187 (Total Wildland Fires and Acres burned from 1960 through 2008, Department of the Interior)</a> </div>
	</td>
</tr>

<tr>
	<td>SPARQL queries</td>
	<td>
	 <div> * <a href="http://data-gov.tw.rpi.edu/demo/stable/demo-1187-40x-wildfire-budget-1187.sparql">http://data-gov.tw.rpi.edu/demo/stable/demo-1187-40x-wildfire-budget-1187.sparql</a> </div>
	</td>
</tr>

<tr>
	<td>SPARQL endpoint</td>
	<td>
	  <a href="http://data-gov.tw.rpi.edu/sparql">http://data-gov.tw.rpi.edu/sparql</a>
	</td>
</tr>
</table>

Wiki Page

wiki page name

create a wiki page, try to make its name the same as the title of the demo page.

   Demo: YOUR_DEMO_NAME
   e.g. http://data-gov.tw.rpi.edu/wiki/Demo:_Linking_Wildland_Fire_and_Government_Budget


wiki page content

You need to fill the following sections

fill thing template, and the following fields are required
  • description
  • image (screenshot)
  • type (use experimental demo)
  • creator
  • created (date)
{{thing
 |type=featured demo
 |description=US government is spending billions of dollars on fighting with Wildland Fire, and this demos show their correlations.
 |creator=Li Ding
 |created=2010/04/08
 |image=File:Demo-wildlandfire-budget.png
}}
fill demo template
  • data source (data.gov datasets)
  • other data source (non-data.gov datasets)
  • ogd (if this demo uses ogd data)
  • live demo (url to demo)
  • technology used
  • sparql (list of sparql)
{{demo
|data source=1187,401
|other data source= Dataset WildfireNews, DBpedia
|ogd=true
|live demo=http://data-gov.tw.rpi.edu/demo/stable/demo-1187-40x-wildfire-budget.html
|technology used=RDF,SPARQL,SparqlProxy,Google Visualization API,TDB Triple Store,Google Visualization Annotated Timeline
|sparql=http://data-gov.tw.rpi.edu/demo/stable/demo-1187-40x-wildfire-budget-1187.sparql
}}
add "Interesting Observations" section
  • show what could be observed through this demo, which specific application area does this demo focuses on, and what problem does this demo try to provide insights.

e.g. Demo:_Linking_Wildland_Fire_and_Government_Budget#Interesting_Observations

add "Technology Highlights" section
  • which technologies are used to address special challenges, visualize what kind of data
  • how can others replicate such technology quickly
  • If your demo uses any new technologies that don't exist in existing demos, you may highlight it.

e.g. Demo:_Linking_Wildland_Fire_and_Government_Budget#Technology_Highlights


 IMPORTANT: After editing your wiki page, please make sure it is free of red link.

Technologies

MIT Exhibit

broken RDF/XML export

We are using exhibit to build demos on US government data. The orange icon (http://api.simile-widgets.org/exhibit/2.2.0/images/liveclipboard-icon.png) allows us to export data in different format. As pointed out by Kingsley, the "RDF/XML" export was broken and always alert a message: "Caught exception: undefined Details: TypeError: D is undefined". I also check the demos at the official Exhibit site, and they also have the same problem.

To fix this issue, to insert the following line after you refer to exhibit-api.js. (thanks for David Huynh)

   <script src="http://data-gov.tw.rpi.edu/2010/exhibit-fix-20100421.js"></script>



Web Browser

broken Google Visualization in Internet Explorer

IE takes strict standard on JSON object parsing. It considers the following JSON array has three elements, while Firefox smartly sees only two elements.

[a,b,]

This minor difference causes major problem with Google visualization: a data row with mismatched column number will cause the visualization not displayable in browser!


SPARQL query

common issues

  • you need to declare namespace for using xsd
PREFIX xsd:  <http://www.w3.org/2001/XMLSchema#> 

Virtuoso "Order By" Problem

It seems that virtuoso does not sort simple named graph query result right if you try the following query on http://data-gov.tw.rpi.edu/sparql, you will see the problem

PREFIX dgp316:<http://data-gov.tw.rpi.edu/vocab/p/316/>
SELECT distinct ?year ?value
WHERE
{GRAPH <http://data-gov.tw.rpi.edu/vocab/Dataset_320>
{
 ?s dgp316:value ?value .
 ?s dgp316:year ?year .
 ?s dgp316:period "M12" .
 ?s dgp316:series_id "CUSR0000SA0" .
}
}
ORDER BY ?year

Datatype Casting

PREFIX dgp32:  <http://data-gov.tw.rpi.edu/vocab/p/32/>
PREFIX xsd:  <http://www.w3.org/2001/XMLSchema#> 
SELECT ?id ?label ?datetime  ?lat ?lon ?magnitude ?depth  ?region ?src ?uri 
FROM <http://data-gov.tw.rpi.edu/raw/34/data-34.rdf>
WHERE { 
         ?uri dgp32:eqid ?id. 
         ?uri dgp32:eqid ?label. 
         ?uri dgp32:region ?region.
         ?uri dgp32:datetime ?datetime. 
	  ?uri dgp32:magnitude ?magnitude.
         ?uri dgp32:depth ?depth. 
         ?uri dgp32:lat ?lat.
         ?uri dgp32:lon ?lon. 
         ?uri dgp32:src ?src.
         filter ( xsd:float(?magnitude) >= 3 && xsd:float(?depth) <= 50 )
}

Aggregation (sum)

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> 
SELECT ?p  sum(xsd:integer (?o)) ?agency
WHERE 
{GRAPH <http://data-gov.tw.rpi.edu/vocab/Dataset_401>
{
 # match the specific BGP first, then filter based on account_name. only join with the completely unbound triple pattern after the filter so that the intermediate result size isn't large.
 {
  ?s 	<http://data-gov.tw.rpi.edu/vocab/p/401/account_name> ?account_name.
  ?s  <http://data-gov.tw.rpi.edu/vocab/p/401/bureau_name> ?bureau.
  ?s <http://data-gov.tw.rpi.edu/vocab/p/401/agency_name> ?agency . 
  filter (regex(?account_name,"Wildland Fire"))
 }
 ?s ?p ?o.
}
}
group by ?p ?agency

Aggregation (count + order by count)

virtuoso and tdb use different syntax. this query is good for virtuoso

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX dgp: <http://data-gov.tw.rpi.edu/whitehouse-gov/dataset/visitor-records/vocab/raw/>
SELECT  ?visitor_first , ?visitor_last ,  count(*) as ?cnt
WHERE
{GRAPH <http://data-gov.tw.rpi.edu/source/whitehouse-gov/dataset/visitor-records/version/2010-Mar-26>
{

?s dgp:visitee_namelast "POTUS" .
?s dgp:namelast ?visitor_last .
?s dgp:namefirst ?visitor_first.

}
}
group by ?visitor_first ?visitor_last
order by desc( ?cnt)
limit 100

The difference is highlighted here

TDB SELECT  ?visitor_first ?visitor_last (count(*) as ?cnt)
Virtuoso SELECT  ?visitor_first , ?visitor_last , count(*) as ?cnt remove parenthesis and use coma to separate
Facts about How to publish and fix data-gov demosRDF feed
Dcterms:created20 April 2010  +
Dcterms:creatorLi Ding  +
Dcterms:descriptionstep by step instructions for publishing data-gov demos. Technology bug fix information is also provided
Dcterms:modified2010-5-17
Dcterms:relationHow to build a data-gov demo  +
Foaf:nameHow to publish and fix data-gov demos
Skos:altLabelHow to publish and fix data-gov demos  +, how to publish and fix data-gov demos  +, and HOW TO PUBLISH AND FIX DATA-GOV DEMOS  +
Personal tools
internal pages