select * from html where url=”http://www.doorstroming.net/index.php/actua/49-qnationalisme-is-nationalismeq.html” and
xpath=’//*[contains(concat( " ", @class, " " ), concat( " ", "MsoNormal", " " ))]‘

Is ugly as sin:

http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20html%20where%20url%3D%22http%3A%2F%2Fwww.doorstroming.net%2Findex.php%2Factua%2F49-qnationalisme-is-nationalismeq.html%22%20and%0A%20%20%20%20%20%20xpath%3D’%2F%2F*%5Bcontains(concat(%20%22%20%22%2C%20%40class%2C%20%22%20%22%20)%2C%20concat(%20%22%20%22%2C%20%22MsoNormal%22%2C%20%22%20%22%20))%5D’&diagnostics=true&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys

Two possible improvements:

Dedicated URL composition template

Create a template that can be called like this:

<xsl:call-template name=”url”>
<xsl:with-param name=”decom”>
<url>
<base>http://query.yahooapis.com/v1/public/yql</base>
<query>
<name>q</name>
<value><xsl:value-of select=”$query”/></value>
</query>
<query>
<name>diagnostics</name>
<value>false</value>
</query>
<query>
<name>env</name>
<value>store://datatables.org/alltableswithkeys</name>
</query>
</url>
</xsl:with-param>
</xsl:call-template>

I’m not elaborating this template here. It should be quite straightforward, but there may be a more generic and even more elegant solution:

String Replacement Template

Using the replace template that was created in the XML Translation project.

<xsl:call-template name=”replace”>
<xsl:with-param name=”string”>
<xsl:text>http://query.yahooapis.com/v1/public/yql?q=$query&diagnostics=true&env=$env</xsl:text>
</xsl:with-param>
<xsl:with-param name=”parameters”>
<query><xsl:value-of select=”encode-for-uri($query)”/></query>
<env><xsl:value-of select=”encode-for-uri(‘store://datatables.org/alltableswithkeys’)”/></env>
</xsl:with-param>
</xsl:call-template>

This is the replace template:

<xsl:template name=”replace”>
<xsl:param name=”string”/>
<xsl:param name=”parameters”/>
<xsl:choose>
<xsl:when test=”count($parameters/*) &gt; 0″>
<xsl:variable name=”string2″>
<xsl:value-of select=”replace($string,concat(‘\$’,name($parameters/*[1])),normalize-space($parameters/*[1]))”/>
</xsl:variable>
<xsl:call-template name=”replace”>
<xsl:with-param name=”string” select=”$string2″/>
<xsl:with-param name=”parameters”>
<xsl:copy-of select=”$parameters/*[position() &gt; 1]“/>
</xsl:with-param>
</xsl:call-template>
</xsl:when>
<xsl:otherwise>
<xsl:value-of select=”$string”/>
</xsl:otherwise>
</xsl:choose>
</xsl:template>

use “http://github.com/vicmortelmans/yql-tables/raw/master/weather/weather.frost.xml”  as weather.frost;select * from weather.frost where location = “Ringsaker”

(note: this location is somewhere in northern Scandinavia, where it’s more likely to freeze

Google Weather API

Google’s secret weather API:

http://www.google.com/ig/api?weather=antwerpen

YQL Open Table Execute

var url = “http://www.google.com/ig/api?weather=$location“;
url = url.replace(/\$location/,encode-uri(location));
xml = y.query(“select * from xml where url=’http://www.google.com/ig/api?weather=antwerpen‘”).results;
var seq = new String();
for each (var lowtemp in xml..forecast_conditions.low.@data) {
seq += (lowtemp > 32 ? ‘+’ : ‘-’);
}
seq = seq.substr(0,3);
var output = new String();
switch (seq) {
case “+++”:
output = “Aanhoudende dooi in $location”;
break;
case “++-”:
output = “Overmorgen gaat het vriezen in $location”;
break;
case “+-+”:
case “+–”:
output = “Morgen gaat het vriezen in $location”;
break;
case “—”:
case “-+-”:
case “–+”:
output = “Aanhoudende vorst in $location”;
break;
case “-++”:
output = “Morgen dooit het in $location”;
break;
}
output = output.replace(/\$location/,location);

YQL XML open table returns no data when strange characters are in the XML

When running a query on the Google weather API for a city in France, very often the returned XML contains strange characters, like e.g. in “Avignon, Provence-Alpes-Côte d’Azur”. When querying for this xml through YQL XML open table, nothing is returned!

The Google API is not providing encoding information:

<?xml version=”1.0″?>

Based on an octal dump, I’d say it’s UTF8:

0000300   s   -   C 303 264 t   e       d   &   #   3   9   ;   A   z

Yahoo weather YQL tables

select * from weather.woeid where w in (select woeid from geo.places where text=”paris”)

Not very useful… only one day forecast, and e.g. for ‘Paris’ returning all cities with the name.

Creating a pipe works better:

http://pipes.yahoo.com/pipes/pipe.run?_id=4f9865c59104e4f105c331c5fbee4116&_render=rss&location=antwerpen

The pipe first gets the woeid for the location using the YQL woeid open table (see above), then loads the weather info from the Yahoo Weather RSS feed.

The result is formatted html, so extracting the low temp values requires text parsing.

World Weather Online API

Registration required, a simple API, but it won’t take ‘Antwerpen’, but only the english name of this flemish city… not so nice!

http://www.worldweatheronline.com/feed/weather.ashx?q=antwerp&format=xml&num_of_days=3&key=81e058dc89161948102509

So let’s use the YQL geo.places open table for getting the coordinates of our location, and feed this to the longitude/latitude World Weather Online API:

http://www.worldweatheronline.com/feed/weather.ashx?q=51.22,4.40&format=xml&num_of_days=3&key=81e058dc89161948102509

This operation chain is performed in this pipe.

This pipe, on it’s turn, can be used as source for YQL open table execute scripting, via the JSON open table query:

select * from json where url=”http://pipes.yahoo.com/pipes/pipe.run?_id=70a400f5404b6a454caf850de3f62d33&_render=json&location=paris” and itemPath=”//tempMinC”

The final weather.frost open table allows for an extra parameter: the target language.

use “http://github.com/vicmortelmans/yql-tables/raw/master/weather/weather.frost.xml”  as weather.frost;select * from weather.frost where location = “Antwerpen” and language=”nl”

Yahoo Pipe

The wrapper pipe is needed to turn the YQL query output into RSS format.

http://pipes.yahoo.com/pipes/pipe.info?_id=145e579bd145fb0b01f9473f69b7e4c1

RSS:

http://pipes.yahoo.com/pipes/pipe.run?_id=145e579bd145fb0b01f9473f69b7e4c1&_render=rss&language=en&location=Paris

encoded:

http%3A%2F%2Fpipes.yahoo.com%2Fpipes%2Fpipe.run%3F_id%3D145e579bd145fb0b01f9473f69b7e4c1%26_render%3Drss%26language%3Den%26location%3DParis

Todo

It would be nice if the language could be automatically detected, based on the provided city name! (but what about Brussels?).

Google Gadget

The original Frost Alert article refers to the Google Webpage where the Google gadget is published.

Managing the site is done in sites.google.com, but that’s OK.

Editing the gadget is done in the iGoogle Gadget Editor. This is the gadget code:

<?xml version=”1.0″ encoding=”UTF-8″?>
<Module>
<ModulePrefs title=”Vorstalarm!” height=”401″ width=”576″ author=”Vic Mortelmans” author_email=”vic.mortelmans@telenet.be”/>
<Content type=”html”><![CDATA[
<script type="text/javascript">
htmlTemplate = '<a href="http://www.addtoany.com/subscribe?linkname=Vorstalarm!&amp;linkurl=http%3A%2F%2Fpipes.yahoo.com%2Fpipes%2Fpipe.run%3F_id%3D145e579bd145fb0b01f9473f69b7e4c1%26_render%3Drss%26language%3Dnl%26location%3D_LOCATION_" target="_blank"><img src="http://static.addtoany.com/buttons/subscribe_120_16.gif" width="120" height="16" border="0" alt="Subscribe"/></a>';
function displayGetRSS () {
var html = htmlTemplate.replace(/_LOCATION_/,document.getElementById('input').value);
document.getElementById('content_div').innerHTML = html;
}
</script>
<div>
<img src="http://farm4.static.flickr.com/3520/4058180670_ed979a4099_o.jpg" width="576" height="401" alt="breughel_hunters" />
<div style="position: relative; background: white; top: -350px; left: 193px; width: 190px; height: 110px; padding: 10px" >
<div style="text-align: center">
<form>
<p>
<b>Woonplaats:</b>
<br/>
<input id="input" name="location" onkeyup="displayGetRSS()" type="text"/>
</p>
</form>
</div>
<div id="content_div" style="text-align: center">
<img src="http://static.addtoany.com/buttons/subscribe_120_16.gif" width="120" height="16" border="0" alt="Subscribe"/>
</div>
</div>
</div>
]]>
</Content>
</Module>

Note that it seems the gagdet must be published (and re-inserted into the webpage) before changes take effect !?

Access via list-based feed

Documentation: http://code.google.com/intl/nl/apis/spreadsheets/data/3.0/reference.html#ListFeed

Example:
https://spreadsheets.google.com/feeds/list/0Au659FdpCliwdG44Q2htMWJEQUxVQ3NfRlZUdlZaalE/1/public/basic
(note: public visibility requires the spreadsheet to be published)
(note: the key, I got when publishing the document)

Result:

<feed>

<title type=‘text’>Blad1</title>

<entry>

<title type=‘text’>christmas</title>

<content type=‘text’>nl: kerstmis, fr: noel</content>

</entry>

<title type=‘text’>easter</title>

<content type=‘text’>nl: pasen, fr: paques</content>

</entry>

</feed>

I have my data, but I’ll have to do some more regex-ing to parse the entries of the feed…

A specific row can be selected using a list feed structured query:
https://spreadsheets.google.com/feeds/list/0Au659FdpCliwdG44Q2htMWJEQUxVQ3NfRlZUdlZaalE/1/public/basic?sq=en%3Dchristmas

Access via Google Visualization API Query Language

Documentation:
http://blog.ouseful.info/2009/05/18/using-google-spreadsheets-as-a-databace-with-the-google-visualisation-api-query-language/
http://code.google.com/intl/nl/apis/visualization/documentation/querylanguage.html

Example:
http://spreadsheets.google.com/a/google.com/tq?key=0Au659FdpCliwdG44Q2htMWJEQUxVQ3NfRlZUdlZaalE
Hmmm… smells like JSON
http://spreadsheets.google.com/a/google.com/tq?key=0Au659FdpCliwdG44Q2htMWJEQUxVQ3NfRlZUdlZaalE&tq=select%20C%20where%20A%20%3D%20′easter’
The column identifier is the column index (‘A’, ‘B’, …) and not the label. That’s a pity. Could be solved by having another table that makes the relationship between the language names and the column indices, but that’s not very clean.

Access via table feed

Documentation:
http://code.google.com/intl/nl/apis/spreadsheets/data/3.0/reference.html#TableFeed

Example:
https://spreadsheets.google.com/feeds/0Au659FdpCliwdG44Q2htMWJEQUxVQ3NfRlZUdlZaalE/tables
https://spreadsheets.google.com/feeds/0Au659FdpCliwdG44Q2htMWJEQUxVQ3NfRlZUdlZaalE/records/0
Nothing there…? This is a bit silly. Seems that you can access the data using this very simple URL-based API, but to create a table, you must use the DATA API and perform a POST request containing the table definition and  with authenticated header.
So setting a ‘range’ in the GUI is not sufficient (as I first assumed).

Further study of the Spreadsheets Data API is needed.

If you are curious about how the Google Data APIs work at the basic level using XML and HTTP, you can read the Protocol Guide. This guide details the requests and responses that the Data API servers expect and return. To learn more about the structure of these requests and responses, read the Reference Guide. This guide defines the API’s feed types, HTTP request parameters, HTTP response codes, and XML elements.

To make working with the API easier, we have a number of client libraries that abstract the API into a language-specific object model. There are Developer’s Guides for Java, .NET, PHP, and Python as well as sample code.

So I guess I’m left with these alternatives:

• building a small java application to create the table
• start fiddling about with http console or curl-like tools to POST the request manually

Let’s not fiddle… And take a tour with the Java client library.

Here  are examples:

• http://code.google.com/p/gdata-java-client/source/browse/trunk/java/sample/spreadsheet/worksheet/WorksheetDemo.java
• http://ga-api-java-samples.googlecode.com/svn/trunk/src/v1/SpreadsheetExporter.java

Client library download (installation is unzipping).

There’s also an Eclipse plugin provided, so I probably should not waste effort using NetBeans.

1. Installation of the plugin
2. TemplateDemo – creating a first project (targeting Spreadsheet rather than Documents)

Got this error: “java.lang.NoClassDefFoundError: com/google/common/collect/Maps”, because I also need the Google Collections (a.k.a Guava).
Installation is unzipping, and adding guava-r06.jar and guava-src-r06.zip to the project’s libraries.

Now the program runs and gives me a list of my spreadsheets:

Getting Spreadsheet entries…

Terminology liturgical days

spiegelserre

krantenwaaier

Lexfeed defects

Kindermissaal

klapluik

multigrade

Total Entries: 7

A few modifications of the example in “create a table” and some browsing through the client’s library javadoc.

// Create a new Spreadsheet service
SpreadsheetService myService = new SpreadsheetService(“My Application”);
myService.setUserCredentials(args[0],args[1]);

TableEntry tableEntry = new TableEntry();

FeedURLFactory factory = FeedURLFactory.getDefault();
URL tableFeedUrl = factory.getTableFeedUrl(“0Au659FdpCliwdG44Q2htMWJEQUxVQ3NfRlZUdlZaalE”);

// Delete table if it already exists
TableFeed tableFeed = myService.getFeed(tableFeedUrl, TableFeed.class);
for (TableEntry table : tableFeed.getEntries()) {
String currTitle = table.getTitle().getPlainText();

if (currTitle.equals(“Terminology”)) {

table.delete();

break;

}

}

// Specify a basic table:
tableEntry.setTitle(new PlainTextConstruct(“Terminology”));
tableEntry.setWorksheet(new Worksheet(“Blad1″));
tableEntry.setHeader(new Header(1));

// Specify columns in the table, start row, number of rows.
Data tableData = new Data();
tableData.setNumberOfRows(200);
// Start row index cannot overlap with header row.
tableData.setStartIndex(2);
// This table has only one column.
tableData.addColumn(new Column(“A”, “en”));
tableData.addColumn(new Column(“B”, “fr”));
tableData.addColumn(new Column(“C”, “nl”));

tableEntry.setData(tableData);
myService.insert(tableFeedUrl, tableEntry);

To change the number of rows (if defined as ‘0’, there are no rows – obviously?), I have to remove the table if it already exists (see bold above).
When retrieving the table feed, I must supply a table number, which seems to increment when tables are deleted and added. So now I can retrieve the data from:
https://spreadsheets.google.com/feeds/0Au659FdpCliwdG44Q2htMWJEQUxVQ3NfRlZUdlZaalE/records/1

But damn me! The records are still printed like “fr: kerstmis, nl: noel, en: christmas”…

A specific record can be retrieved using a structured query:
https://spreadsheets.google.com/feeds/0Au659FdpCliwdG44Q2htMWJEQUxVQ3NfRlZUdlZaalE/records/1?sq=en%3Dchristmas

Cell based feed

First I need the worksheets feed

https://spreadsheets.google.com/feeds/worksheets/0Au659FdpCliwdG44Q2htMWJEQUxVQ3NfRlZUdlZaalE/private/full

In the returned feed, this tag can be found:

<id>https://spreadsheets.google.com/feeds/worksheets/0Au659FdpCliwdG44Q2htMWJEQUxVQ3NfRlZUdlZaalE/private/full/od6</id>

The ‘od6’ is the ID of the worksheet, to be used in the cell based feed URL:

https://spreadsheets.google.com/feeds/cells/0Au659FdpCliwdG44Q2htMWJEQUxVQ3NfRlZUdlZaalE/od6/private/full

The return contains individual cells:

A1

dinsdag 21 september 2010 21:36

en

B1

dinsdag 21 september 2010 21:36

fr

C1

dinsdag 21 september 2010 21:36

nl

Or as xml:

<feed>
<entry>
<title type=‘text’>A1</title>
<content type=‘text’>en</content>
<gs:cell row=’1′ col=’1′ inputValue=‘en’>en</gs:cell>
</entry>
<entry>
<title type=‘text’>B1</title>
<content type=‘text’>fr</content>
<gs:cell row=’1′ col=’2′ inputValue=‘fr’>fr</gs:cell>
</entry>
<entry>
<title type=‘text’>C1</title>
<content type=‘text’>nl</content>
<gs:cell row=’1′ col=’3′ inputValue=‘nl’>nl</gs:cell>
</entry>
<entry>
<title type=‘text’>A2</title>
<content type=‘text’>christmas</content>
<gs:cell row=’2′ col=’1′ inputValue=‘christmas’>christmas</gs:cell>
</entry>
<entry>
<content type=‘text’>kerstmis</content>
<gs:cell row=’2′ col=’2′ inputValue=‘kerstmis’>kerstmis</gs:cell>
</entry>
<entry>
<title type=‘text’>C2</title>
<content type=‘text’>noel</content>
<gs:cell row=’2′ col=’3′ inputValue=‘noel’>noel</gs:cell>
</entry>
<entry>
<title type=‘text’>A3</title>
<content type=‘text’>easter</content>
<gs:cell row=’3′ col=’1′ inputValue=‘easter’>easter</gs:cell>
</entry>
<entry>
<title type=‘text’>B3</title>
<content type=‘text’>pasen</content>
<gs:cell row=’3′ col=’2′ inputValue=‘pasen’>pasen</gs:cell>
</entry>
<entry>
<title type=‘text’>C3</title>
<content type=‘text’>paques</content>
<gs:cell row=’3′ col=’3′ inputValue=‘paques’>paques</gs:cell>
</entry>
</feed>

This should be queriable by a clever XPath via YQL.

feed/entry[gs:col/@col = (<column where first row is $language>) and gs:row/@row = (row where first column is $text)]/content

/feed/entry[gs:col/@col = (/feed/entry[gs:cell/@row = ‘1’ and content = $language]/@col) and gs:row/@row = (/feed/entry[gs:cell/@col = ‘A’ and content=$text]/@row)]/content

But -alas!- there seems to be an issue. The XML table isn’t returning any data at all:

select * from xml where url=”https://spreadsheets.google.com/feeds/cells/0Au659FdpCliwdG44Q2htMWJEQUxVQ3NfRlZUdlZaalE/od6/private/full”

Even after putting https in front of the yql URL… (I suspect it’s because the google feed URL is on https)…

> The explanation is easy: this feed requires authentication…

>> Wait a minute… after reading this, I just did ‘Publish as webpage’ and selected ‘Atom’ and ‘Cells’ as options, and got this URL:

https://spreadsheets.google.com/feeds/cells/0Au659FdpCliwdG44Q2htMWJEQUxVQ3NfRlZUdlZaalE/od6/public/basic

This URL is public, so it works in YQL !!

select * from xml where url=”https://spreadsheets.google.com/feeds/cells/0Au659FdpCliwdG44Q2htMWJEQUxVQ3NfRlZUdlZaalE/od6/public/basic”

I can also get the data in CSV format (been reading this and this)

https://spreadsheets.google.com/pub?key=0Au659FdpCliwdG44Q2htMWJEQUxVQ3NfRlZUdlZaalE&hl=nl&single=true&gid=0&output=csv

And perform this query on the CSV open table,

select * from csv where url=”https://spreadsheets.google.com/pub?key=0Au659FdpCliwdG44Q2htMWJEQUxVQ3NfRlZUdlZaalE&hl=nl&single=true&gid=0&output=csv”

which on it’s turn can be fed into the XML table….
- or I should be able to add to the query ‘and columns = “en,fr,nl” and en=”christmas”’, the column names list being whatever is appropriate for the queried Spreadsheet.

Or wait,

1. publish the spreadsheet as CSV
2. turn the CSV into XML using YQL CSV open table
   REST URL
3. query the XML using YQL XML table

The query:

select * from xml where itemPath = “/query/results/row/*[../*[name(.) = name(../../row[1]/*[. = 'en'])] = ‘easter‘ and name(.) = name(../../row[1]/*[. = 'nl'])]” and  url = “http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20csv%20where%20url%3D%22https%3A%2F%2Fspreadsheets.google.com%2Fpub%3Fkey%3D0Au659FdpCliwdG44Q2htMWJEQUxVQ3NfRlZUdlZaalE%26hl%3Dnl%26single%3Dtrue%26gid%3D0%26output%3Dcsv%22&diagnostics=false”

This is equivalent to:

select nl from key where en = ‘easter‘

It’s fairly simple to mimick (some of) the behaviour of Dapper using these two tools. I’m into this, because I bumped into a Dapper disadvantage (well, in most cases it’s an advantage, but not in my particular case): it reduces the captured html fragments to plain text, which is good for a simple RSS update, but bad if you want to retain formatting in captured text (e.g. italics may be important and relevant for reflecting the content of a text).

Procedure:

1. Install the SelectorGadget in your browser (it’s a bookmarklet, so you can simply drag it into your bookmarks)
2. Open the webpage you want to retrieve data from (or at least a very similar webpage)
3. Start the SelectorGadget
4. Play around until you’ve selected the right sections
5. Copy the displayed CSS selector
6. Open the YQL console
   You’ll see a sample queryselect * from data.html.cssselect where url=”http://www.doorstroming.net/index.php/actua/49-qnationalisme-is-nationalismeq.html” and css=”.MsoNormal”
7. Replace the value for ‘css’ with the CSS selector generated by SelectorGadget
8. Replace the value for ‘url’ with the URL of the page containing your data. This can be any page with a similar structure to the page you’ve used originally.
9. Execute the query or copy the REST URL

The REST URL looks like this:
http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20data.html.cssselect%20where%20url%3D%22http%3A%2F%2Fwww.doorstroming.net%2Findex.php%2Factua%2F49-qnationalisme-is-nationalismeq.html%22%20and%20css%3D%22.MsoNormal%22&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys
By replacing the bold sections with resp. the (url-encoded) URL of the website and the applicable CSS Selector (also url-encoded!), you can apply any query on any webpage.

Restrictions:

• for more complex selections, SelectorGadget returns stuff like this
  :nth-child(3) , .article-meta
  It looks like the YQL table can’t cope with this (probably the :nth-child() is something coming from a newer CSS standard than supported by the YQL table)
• Dapper can do lots more, like capturing multiple fields, grouping, rendering RSS and other output formats,…
• Dapper has an intuitive GUI

Using html open table

To solve the first restriction, using the html open table is a good solution. SelectorGadget has a button for converting the CSS selector into an XPath expression. The query then looks like this:
select * from html where url=”http://www.doorstroming.net/index.php/actua/49-qnationalisme-is-nationalismeq.html” and xpath=’//*[contains(concat( " ", @class, " " ), concat( " ", "MsoNormal", " " ))]‘

I spent a couple of days figuring out some weird behaviour of Serna. When trying to develop my own plugin, it always rendered my content in editable text fields, rather than word-processor-like inline editable text. Looking at the FO-tree dump, it appeared to be rendered using the serna-extensions.

Strip-down

I decided to strip-down the resume project, up to the level at which my own plugin was built up, to see where the text fields would show up.

The trigger for getting the editable text fields, is removing this line from the source resume xml:

<!DOCTYPE resume PUBLIC "-//Sean Kelly//DTD Resume  1.5.1//EN" "http://xmlresume.sourceforge.net/dtd/resume.dtd" []>

A clue on the web?:

It’s not a bug  per se.  That final decision rests with the folks at Syntext.  Serna has the capability to support entities (which XML Schema does not have).  In order to do get that capability, Serna uses in-line DOCTYPE.

That said, to process those documents in the Toolkit one needs to do one of the following:

a) remove the DOCTYPE from the XML docs; you get XSD validation

b) remove xmlxs:xsi namespace and xsi:noNamespaceSchemaLocation attribute from the XML docs;  you get DTD validation

c) modify the DTDs in the Toolkit  to include xmlxs:xsi namespace and xsi:noNamespaceSchemaLocation attribute;  you get DTD validation

Changing the XML parser parameters will not do anything useful since it is dependent on what appears in the XML docs.  The easiest one to do ay this point would be to modify the DTDs in the Toolkit.  That would allow you to process the documents without having to modify each one them.

Further stripping down of the resume example shows how the DTD declares a FIXED xmlns-attribute to the top-level element. Setting or unsetting this attribute would cause Serna to behave differently regarding presentation of the element text in wysiwyg-view. After removing the DTD alltogether, the xmlns-attribute doesn’t seem to be needed at all.

Bare-bone Serna plug-in

This is the complete specification of a bare-bone Serna plug-in that can be used for displaying, editing and saving xml-files using a custom stylesheet.

Serna Document Template (sdt)

<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE  t:document-template PUBLIC "-//Syntext//DTD Syntext Document Template  V2.0//EN" "doctemplate2.dtd">
<t:document-template  xmlns:t="http://www.syntext.com/Extensions/DocTemplate-2.0">
 <t:name>Resume</t:name>
 <t:category>XML  Resume</t:category>
 <t:xml-stylesheet>$SERNA_TEMPLATE_DIR/xsl/resume.xsl</t:xml-stylesheet>
 <t:xml-schema>$SERNA_TEMPLATE_DIR/xsd/resumex.xsd</t:xml-schema>
 <t:root-elements>resume</t:root-elements>
 <t:category-pixmap>resume_1_5_1_template</t:category-pixmap>
 <t:rootelem-guess-string>resume</t:rootelem-guess-string>
 <t:document-skeleton>
  <t:name>Resume</t:name>
 </t:document-skeleton>
</t:document-template>

XML Schema (resumex.xsd)

<?xml  version="1.0"?>
<xs:schema  xmlns:xs="http://www.w3.org/2001/XMLSchema">
 <xs:element  name="misc">
  <xs:complexType>
   <xs:sequence>
    <xs:element ref="para"   maxOccurs="unbounded"/>
   </xs:sequence>
  </xs:complexType>
 </xs:element>
 <xs:element  name="para">
  <xs:complexType mixed="true" />
 </xs:element>
 <xs:element  name="resume">
  <xs:complexType>
   <xs:sequence>
    <xs:element ref="misc"   maxOccurs="unbounded"/>
   </xs:sequence>
  </xs:complexType>
 </xs:element>
</xs:schema>

Stylesheet (resume.xsl)

<xsl:stylesheet  version="1.0"
 xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
 xmlns:fo="http://www.w3.org/1999/XSL/Format">

<xsl:include  href="http://www.syntext.com/xslbricks-1.0/fo/fo.xsl"/>

<xsl:strip-space  elements="*"/>

<xsl:template name="heading">
 <fo:block>
  <xsl:value-of  select="local-name(.)"/>
 </fo:block>
</xsl:template>

<xsl:template  match="para">
 <fo:block>
  <xsl:call-template name = "bold.inline"/>
 </fo:block>
</xsl:template>

<xsl:template  match="misc">
 <fo:block>
 <xsl:call-template name="heading"/>
 <xsl:apply-templates/>
 </fo:block>
</xsl:template>

<xsl:template  match="resume">
 <xsl:call-template name="div"/>
</xsl:template>

</xsl:stylesheet>

Example XML

<?xml version='1.0'  encoding='UTF-8'?>
<resume>
 <misc>
  <para>User  Documentation developer</para>
  <para>Strip-downer</para>
 </misc>
</resume>

Conclusion

I seem to have solved my problem, but I still don’t understand why Serna behaves the way it does. Anyway, lesson learned (once again): if a stylesheet is behaving strange, 9 to 10 a namespace is in your way!

[Update!] Conclusion 2

I must have been completely deceived…. now it seems more like that the element definition in the xsd controls the behaviour of Serna:
This definition renders text fields:

<xs:element name="hyperlink"  type="xs:string" minOccurs="0" maxOccurs="1"/>

And omitting the ‘type=”xs:string”’ renders wordprocessor-style inline text:

<xs:element  name="title" minOccurs="0" maxOccurs="1"/>

Check the documentation and the development notes for more details.

You want the get the contents of something that is only a reference:

Perl doesn’t (ever) implicitly dereference your variables. If you want to get at the thing a reference is referring to, then you have to do this yourself using either prefix typing indicators, like ${$blah}, @{$blah}, @{$blah[$i]}, or else postfix pointer arrows, like $a->[3], $h->{fred}, or even $ob->method()->[3].

You want to copy the contents of something that is only a reference:

In fact, except for passing arguments to functions, I seldom like to see the gimme-a-reference operator (backslash) used much at all in code. Instead, I advise beginners that they (and most of the rest of us) should try to use the much more easily understood constructors [] and {} instead of relying upon lexical (or dynamic) scoping and hidden reference-counting to do the right thing behind the scenes.

here’s a report on a quest in persuit of harvesting data from multi-page websites. Handling single pages is trivial, using tools like Dapper or YQL. It’s becoming tricky when targeting data that’s stored on separate webpages, like e.g. this one:

http://www.kerknet.be/zoek_parochie.php?allbisdom=1

The complete list of parishes is split over 7 separate pages, and I’d like to have them all together.

Before diving into the series of attempts, first let’s distinct two typicaly situations:
1) split data; the data is split over multiple pages and the pages are sequentially linked together
2) deep data; the main  page only contains links to the pages where the actual data is available

The above example is actually a combination of both, with as special factor that each datapage contains a single datarecord with different datafields.

Deep data

First attempt: YQL (failed)

This YQL query fetches the links on a single page:

select href from html where url = “http://www.kerknet.be/zoek_parochie.php?allbisdom=1″ and xpath = “//table[@class='parochies']/tr/td[@class='col3']/a”

Results are like this:

<results>
<a href=”/parochie/parochie_fiche.php?parochieID=24″/>
<a href=”/parochie/parochie_fiche.php?parochieID=21″/>
<a href=”/parochie/parochie_fiche.php?parochieID=22″/>
…
</results>

Now I thought that the result of this query could be used as subquery for providing the actual dataquerey the list of pages using the “url in ()” statement, but overlooked the fact that the retrieved urls are relative.

So this YQL query is not working at all:

select * from html where xpath = “//table[@class='contact first']” and url in (select href from html where url = “http://www.kerknet.be/zoek_parochie.php?allbisdom=1″ and xpath = “//table[@class='parochies']/tr/td[@class='col3']/a”)

Second attempt: Dapper + YQL (succes on deep data issue)

Now I created a Dapper to extract the links from the overview page. Luckily, Dapper returns full paths, because it’s designed to render RSS feeds. The Dapper is configured such that it returns xml data-items such as this:

<item dataType=”RawString” fieldName=”item” href=”http://www.kerknet.be/parochie/parochie_fiche.php?parochieID=65″ originalElement=”a” type=”field”>SINT-ANNA</item>

The output of the Dapper can now be queried by YQL:

select item.href from xml where url = “http://www.dapper.net/RunDapp?dappName=KerknetParochiesdeephtml&v=1&applyToUrl=http%3A%2F%2Fwww.kerknet.be%2Fzoek_parochie.php%3Fallbisdom%3D1″

… and submitted to the “url in ()” statement:

select * from html where xpath = “//table[@class='contact first']” and url in (select item.href from xml where url = “http://www.dapper.net/RunDapp?dappName=KerknetParochiesdeephtml&v=1&applyToUrl=http%3A%2F%2Fwww.kerknet.be%2Fzoek_parochie.php%3Fallbisdom%3D1″)

This works, but it’s not addressing the split data on the ‘next’ list pages.

Third attempt: Yahoo Pipes + Dapper (success on split data issue)

YQL cannot solve the split data issue without creating a dedicated table and do some javascript coding. This table might have an interface looking like this:

select * from deepdata
where url = “http://www.kerknet.be/zoek_parochie.php?allbisdom=1″
and urlxpath = “//table[@class='parochies']/tr/td[@class='col3']/a/@href”
and nextxpath = “//span[@class='pagelinks']/a[current()="Volgende"]/@href”
and xpath = “//table[@class='contact first']“

Where:

But as proof of concept, my next attempt involves Yahoo Pipes.

Conceptually, for retrieving each next page, a recursive setup would be ideal. A pipe cannot be called recursively out of the box, but I found the answer here:

Note that, in Yahoo! Pipes, recursion is performed by calling the pipe.run method of the <pipes.yahoo.com> service with the unique ID of the Pipe, parameters, and desired output format (e.g., JSON). Attempting to include a Pipe as a subpipe to itself directly may work while using the Pipes Editor, but does not when the Pipe is run outside the editor.

And after (quite) a while, here’s my pipe that returns the dapper result for the provided page + the dapper results for the next pages, by calling itself recursively. Note that it doesn’t do the actual deep data retrieval yet !

http://pipes.yahoo.com/pipes/pipe.edit?_id=3f5c8fc808ae3f27b3d6a5b578aa0da0

These are it’s input fields:

An example of running the pipe for rendering as rss:

http://pipes.yahoo.com/pipes/pipe.run?ListDapper=KerknetParochiesdeephtml&ListPage=http%3A%2F%2Fwww.kerknet.be%2Fzoek_parochie.php%3Fallbisdom%3D1&NextDapper=KerknetParochiesdeephtmlnext&_id=3f5c8fc808ae3f27b3d6a5b578aa0da0&_render=rss

Note that it takes easily up to 2 minutes for the pipe to return! That’s for accessing 7 pages and returing a feed of ~300 items. What to expect when in the end ~300 pages must be downloaded and analyzed !!

Some notes that I made during debugging:

• for debugging, using the pipe description page is giving better debug info than the development screen!
• pipes are meant for processing rss data; generic xml data is bound to give errors.

Fourth attempt: Yahoo Pipes + Dapper + YQL (failed)

Combination of attempts 2 and 3 gives this:

select * from html where xpath = “//table[@class='contact first']” and url in (select link from feed where url = “http://pipes.yahoo.com/pipes/pipe.run?ListDapper=KerknetParochiesdeephtml&ListPage=http%3A%2F%2Fwww.kerknet.be%2Fzoek_parochie.php%3Fallbisdom%3D1&NextDapper=KerknetParochiesdeephtmlnext&_id=3f5c8fc808ae3f27b3d6a5b578aa0da0&_render=rss
“)

The main downside is that YQL returns messy extracts from (as messy) html code. So either a dedicated YQL table containing javascript code should be deviced, or dapper is called to the rescue again. Prepare for really bad performance this time!

Pipes won’t managing the dapper output… it has a hard time managing the XML data outside RSS context.

Fifth attempt: Yahoo Pipes + Dapper + YQL (success, but…)

Attacking the problem the other way around, the output of the messy YQL query can be submitted to dapper for extracting the data in a proper format.

The rest url becomes unreadible, but for sake of reference, it’s here:

http://www.dapper.net/RunDapp?dappName=KerknetParochiespipe&v=1&applyToUrl=http%3A%2F%2Fquery.yahooapis.com%2Fv1%2Fpublic%2Fyql%3Fq%3Dselect%2520*%2520from%2520html%2520where%2520xpath%2520%3D%2520%2522%2F%2Ftable%255b%40class%3D%27contact%2520first%27%255d%2522%2520and%2520url%2520in%2520%28select%2520link%2520from%2520feed%2520where%2520url%2520%3D%2520%2522http%3A%2F%2Fpipes.yahoo.com%2Fpipes%2Fpipe.run%3FListDapper%3DKerknetParochiesdeephtml%26ListPage%3Dhttp%253A%252F%252Fwww.kerknet.be%252Fzoek_parochie.php%253Fallbisdom%253D1%26NextDapper%3DKerknetParochiesdeephtmlnext%26_id%3D3f5c8fc808ae3f27b3d6a5b578aa0da0%26_render%3Drss%2522%29%26diagnostics%3Dtrue

The main problem for debugging is that all these services use thorough caching and on top, dapper is quite unreliable. So every once in a while, either you’ve made a typo, or dapper is refusing service, and somewhere in the chain, you end up with faulty data, which is cached and re-appearing on subsequent calls.

Also, performance of this setup is really bad. It could be used for offline data-retrieval, but not for interactive systems.

Sixth attempt: Same ingredients in different order

So I’ll use

1. Yahoo Pipes because it has a clear interface to the end-user; my only fear is still that it won’t output custom xml formats
2. YQL because it allows to program part of the logic and because it’s very performant
3. Dapper because it has an excellent combination of logical data-extraction power and user-friendlyness

The pipe will pick up the parameters and call the YQL query. The query calls the dappers and collects the data.

Parameters:

This is the “split deep data” scenario query logic in pseudo-code:

list = yql(select * from xml where url=”http://www.dapper.net/dapper=listdapper&url=url”)
foreach dataurl in list {

newdata = yql(select * from xml where url=”http://www.dapper.net/dapper=datadapper&url=dataurl”)

data = concatenate(data, newdata)

}
nexturl = yql(select * from xml where url=”http://www.dapper.net/dapper=nextdapper&url=url”)
if next {

nextdata = yql(select * from splitdeepdata where datadapper=datadapper and listdapper=listdapper and nextdapper=nextdapper and url=nexturl)

data = concatenate(data, nextdata)

}
return data

Listdapper

select href, content from xml where url=”http://www.dapper.net/RunDapp?dappName=kerknetparochieslist&v=1&applyToUrl=http%3A%2F%2Fwww.kerknet.be%2Fzoek_parochie.php%3Fallbisdom%3D1″ and itemPath=”//item”

returns

<results>
<item href=”http://www.kerknet.be/parochie/parochie_fiche.php?parochieID=24″>O.-L.-VROUW TEN HEMEL OPGENOMEN</item>
<item href=”http://www.kerknet.be/parochie/parochie_fiche.php?parochieID=21″>SINT-ANDRIES</item>
…
</results>

note: the dapper must be configured to return hyperlinks to the datapages as ‘item’ elements

Datadapper

select * from xml where itemPath=”//item” and url=”http://www.dapper.net/RunDapp?dappName=KerknetParochiesdata&applyToUrl=http%3A%2F%2Fwww.kerknet.be%2Fparochie%2Fparochie_fiche.php%3FparochieID%3D24&v=1″

returns

<results>
<item groupName=”item” type=”group”>
<parochie dataType=”RawString” fieldName=”parochie” originalElement=”th” type=”field”>O.-L.-VROUW TEN HEMEL OPGENOMEN, ANTWERPEN</parochie>
<heiligemis dataType=”RawString” fieldName=”heiligemis” originalElement=”th” type=”field”>Za</heiligemis>
<heiligemis dataType=”RawString” fieldName=”heiligemis” originalElement=”td” type=”field”>16.00u Eucharistieviering 17.00u Eucharistieviering</heiligemis>
<heiligemis dataType=”RawString” fieldName=”heiligemis” originalElement=”th” type=”field”>Zo</heiligemis>
<heiligemis dataType=”RawString” fieldName=”heiligemis” originalElement=”td” type=”field”>09.00u Eucharistieviering 10.30u Eucharistieviering 12.00u Eucharistieviering 17.00u Eucharistieviering</heiligemis>
<email dataType=”RawString” fieldName=”email” href=”http://www.kerknet.be/parochie/mailto.php?parochieID=24&amp;mailto=92″ originalElement=”a” type=”field”>E-mail</email>
<adres1 dataType=”RawString” fieldName=”adres1″ originalElement=”span” type=”field”>Sint-Pieterstraat 1</adres1>
<adres2 dataType=”RawString” fieldName=”adres2″ originalElement=”span” type=”field”>2000 Antwerpen</adres2>
<telefoon dataType=”RawString” fieldName=”telefoon” originalElement=”span” type=”field”>Tel. 03/213.99.60</telefoon>
</item>
</results>

note: the dapper must be configured such that it groups elements in groups named ‘item’ !

Trying it out (see the open table logic on git):

use ‘http://github.com/vicmortelmans/yql-tables/raw/master/data/deepdapper.xml’ as deepdapper;
select * from deepdapper where url = “http://www.kerknet.be/zoek_parochie.php?allbisdom=1″
and datadapper = “KerknetParochiesdata”
and nextdapper = “KerknetParochiesdeephtmlnext”
and listdapper = “kerknetparochieslist”

Damn! bumping already into YQL’s execution rate limits:

The following rate limits apply to executions within Open Data Tables:

Hey! it shouldn’t be giving an error sounding like this:

Exception: Circular table reference detected while using ‘deepdapper’. This table was already used in the call stack deepdapper

So, can’t I do recursion in YQL? – No, I can’t. Drawing my conclusions:

My attempt was to scrape data from paged html content. The table would scrape the content from the first page (the one provided in the where clause), then go looking for a ‘next page’ hyperlink and call itself again providing the next page url in the where clause, and return the collected data from the first page bundled with the data collected by the recursive call.

I guess I may have to look into the Open Tables paging functionality, or rewrite the algorithm as a loop.

Maybe the pagination feature should bring a fix? But I doubt.

Seventh attempt: rewriting the open table as a loop

next = url
while next {

list = yql(select * from xml where url=”http://www.dapper.net/dapper=listdapper&url=next”)

foreach dataurl in list {

newdata = yql(select * from xml where url=”http://www.dapper.net/dapper=datadapper&url=dataurl”)

data = concatenate(data, newdata)

}

next = yql(select * from xml where url=”http://www.dapper.net/dapper=nextdapper&url=next”)

}
return data

Still debugging, but read something interesting: “Queries called from y.query return and execute instantly. However, data is only returned when the results property is accessed. This feature allows you to make multiple, independent queries simultaneously that are then allowed to process before being returned together when the results property is accessed.” So  now I’m collecting all dataqueries in an array and only at the end, reading out the results… or maybe this isn’t so good at all, because now dapper will be attacked with the queries almost instantanously.

More debugging notes:

• When I concatenate two XML lists, it seems to become a string !? The concatenation operator ‘+’ or ‘+=’ doesn’t work properly. Using the ‘appendChild’ method is safer!
• Seems like submitting an XML List as response.object also isn’t a good idea. It likes genuine XML objects better!

Success!!

Now also making nextdapper and listdapper optional, and adding nextcount to limit the number of pages… and trying to find a better example than Kerknet for the documentation.

OK, this is ready for release. Documented and all!

Appendix A – filtering results of YQL xml table

In the itemPath parameter (not called ‘xpath’, as in the html table!), you can filter elements from the raw xml. Filtering down to attribute level won’t do anything.

In the select statement, you can filter elements down the result tree. Use ‘.’-separated paths. Don’t mention the root element! You can also address attributes (without ‘@’ !). Use ‘content’ to refer to the content of an element. List multiple path specifications to cover more data. The structure of the result tree remains the same (this means that attributes remain attributes!), but it’s limited to the elements the match the specified path(s), their parents and children.

Appendix B – small Git reminder notes

git add bible.xml
git commit -m “adding open table containing bible passages”
git push origin master

git pull
/*after editing online*/

Appendix C – debugging YQL tables

Yahoo! http://developer.yahoo.com/yql/console/?debug=true

I wrote a similar program before, in C, making calls to the ‘pstools’ command-line toolset. That wasn’t very flexible and not portable at all. The only alternative program on the web is Quantum Elephant’s Bookbinder. I couldn’t get it to work on some PDF’s, but it has as a great advantage that it’s relatively easy to install and that it has a GUI (how many bookbinders will know how to install Perl and run a command-line script?).

Perl has a couple of PDF libraries at CPAN to work with PDF. Signaturize uses PDF::API2. The documentation for this library is very, very minimalistic, so I hope that by making signaturize public, it can be of use for anyone who’s going to use PDF::API2, even more than for people with a passion for bookbinding .

The PDF::API2 api is very simple, but one have to know the magic sequence of commands to do the trick…

First, create a new pdf object and populate it with some pages. The page size must be defined for each separate page.

my $outPdf = PDF::API2->new;

my @outPage;

for ($nr=1; $nr<=$outPages; $nr++) {

 $outPage[$nr] = $outPdf->page;

 $outPage[$nr]->mediabox($outXMediaSize, $outYMediaSize);

}

Now you can start to work. For example, drawing a dashed line on one of the pages:

my $line = $outPage[$nr]->gfx;

$line->save;

$line->linedash((5,15));

$line->move($startx,$starty);

$line->line($stopx,$stopy);

$line->stroke;

$line->restore;

Or you can write some text:

my %font = (

 Helvetica => {

 Bold => $outPdf->corefont( 'Helvetica-Bold', -encoding => 'latin1' ),

 Roman => $outPdf->corefont( 'Helvetica', -encoding => 'latin1' ),

 Italic => $outPdf->corefont( 'Helvetica-Oblique', -encoding => 'latin1' ),

 },

);

my $text = $outPage[$nr]->text;

$text->save;

$text->font($font{'Helvetica'}{'Roman'}, 0.2/cm );

$text->translate($x,$y);

$text->text("Hallo");

$text->restore;

Or you can copy the contents of some page from another PDF:

my $insertedpageXobject = $outPdf->importPageIntoForm($inPdf, $nr); #fetch the page as Xobject

my $insertedpageGfx = $outPage[$outPageNr]->gfx; #create a new graphics object

$insertedpageGfx->save; #otherwise the previous tranformation is added

$insertedpageGfx->transform(

 -translate => [$x,$y],

 -rotate => $rot,

 -scale => [$scale,$scale],

); #define a transformation on the graphics object

$insertedpageGfx->formimage($insertedpageXobject, 0, 0, 1); #drop the page Xobject in the graphics object

$insertedpageGfx->restore;

There’s another great hands-on tutorial here, where you’ll learn how to write body text and do more graphical stuff.

What took me very long was to understand the behaviour of the gfx objects and text objects. They have a common methodset that’s part of PDF::API2::Content. To my feeling, the gfx-object is attached to the page, and each page only has got one gfx object, so calling the gfx-method multiple times on a single page doesn’t make sense. The behaviour of the save and restore methods is not intuitive at all, but I learnt to always call ‘save’ before starting a drawing operation and ‘restore’ after finishing it. If you don’t, you’ll see that transformation behaviour is carried over to subsequent drawing operations, or that weird ‘restore’ errors are displayed when reading the PDF.

Nah. It’s useful as an experiment! The service is provided on a google site that mainly consists of a single google gadget. The gadget asks the users where he’s living and returns an RSS feed, which he can subscribe to using any RSS feed reader or alert system available. The RSS feed on it’s turn, is generated by a yahoo pipe that uses dapper to parse the weather forecast that’s provided by google web search.

What’s the use of this all… not much really, because an online service like this is completely unreliable. If one of the component services changes specs or even stops working, the whole setup falls apart (and in one frosty night, nation-wide crops will be lost, because the farmers were not alarmed ).

[20100925] check for an update on this article here.