Upgrade Notes

Users new to DataSource can skip this section and proceed directly to the Getting Started section. Implementers who are upgrading from previous versions of DataSource should note the following changes as of version 2.6.0:

• The DataSource class has been refactored into a DataSourceBase base class and the subclasses LocalDataSource, FunctionDataSource, XHRDataSource, and ScriptNodeDataSource. While backward compatibility of the YAHOO.util.DataSource constructor has been maintained, implementers should be aware that calling new YAHOO.util.DataSource() now actually returns one of these subclasses. Implementers can alternatively call a subclass constructor directly. The DataSource constructor returns one of the subclasses based on the oLiveData passed to it, or the dataType config value. This class-based architecture no longer meaningfully supports swapping data types on the fly.
• Parsing of totalRecords is no longer supported as a top-leval schema value. Implementers should access this data as a metaFields value.

Getting Started

The DataSource code must be included before any control can use it:

<!-- Dependencies -->
<script src="http://yui.yahooapis.com/2.9.0/build/yahoo/yahoo-min.js"></script>
<script src="http://yui.yahooapis.com/2.9.0/build/event/event-min.js"></script>
 
<!-- OPTIONAL: JSON Utility -->
<script src="http://yui.yahooapis.com/2.9.0/build/json/json-min.js"></script>
 
<!-- OPTIONAL: Connection (enables XHR) -->
<script src="http://yui.yahooapis.com/2.9.0/build/connection/connection-min.js"></script>
 
<!-- Source file -->
<!-- OPTIONAL: Get (enables dynamic script nodes) -->
<script src="http://yui.yahooapis.com/2.9.0/build/get/get-min.js"></script>
 
<script src="http://yui.yahooapis.com/2.9.0/build/datasource/datasource-min.js"></script>
 

<!-- Dependencies -->
<script src="http://yui.yahooapis.com/2.9.0/build/yahoo/yahoo-min.js"></script>
<script src="http://yui.yahooapis.com/2.9.0/build/event/event-min.js"></script>
 
<!-- OPTIONAL: JSON Utility -->
<script src="http://yui.yahooapis.com/2.9.0/build/json/json-min.js"></script>
 
<!-- OPTIONAL: Connection (enables XHR) -->
<script src="http://yui.yahooapis.com/2.9.0/build/connection/connection-min.js"></script>
 
<!-- Source file -->
<!-- OPTIONAL: Get (enables dynamic script nodes) -->
<script src="http://yui.yahooapis.com/2.9.0/build/get/get-min.js"></script>
 
<script src="http://yui.yahooapis.com/2.9.0/build/datasource/datasource-min.js"></script>
 

Overview

A DataSource is an abstract representation of a live set of data that presents a common predictable API for other objects to interact with. The nature of your data, its quantity, its complexity, and the logic for returning query results all play a role in determining your type of DataSource. For small amounts of simple textual data, a JavaScript array is a good choice. If your data has a small footprint but requires a simple computational or transformational filter before being displayed, a JavaScript function may be the right approach. For very large datasets — for example, a robust relational database — or to access a third-party webservice you'll certainly need to leverage the power of a Script Node or XHR DataSource. Whatever the source of your live data, the following diagram describes an overview of the DataSource API:

1. A request for data is made to the DataSource by a widget instance. oRequest may define specific parameters, or it may simply be null. The sendRequest method requires two arguments:

   oRequest

   For remote data, this request may be a param/value string, such as "?id=123&name=foo". For local data, this request maybe a simpler value such as 123. Specifying parameters may be irrelevent, so this value may be simply be null

   oCallback

   An object literal with the following properties:

   oCallback properties

   Property	Type	Description
   success	Function	The function to call when the response is ready.
   failure	Function	The function to call upon a response failure condition.
   scope	Widget instance	The object to serve as the scope for the success and failure handlers.
   argument	Object	Arbitrary data to be passed back to the success and failure handlers.

2. The DataSource checks its local cache for a response to the oRequest
3. Not finding a cached response will cause the DataSource to make a connection to the live data. Depending on the dataType, the connection may make a remote connection via Connection Manager or the Get Utility, or simply access local data in thread.
4. A response from the live data set is returned to the DataSource in its raw state.
5. The doBeforeParseData() abstract method allows implementers to access and transform the full response before it is schema-parsed.
6. The DataSource parses the raw response according to responseType and implementer-defined schema.
7. The doBeforeCallback() abstract method allows implementers to access and transform the parsed response before it is cached.
8. The parsed data is returned to the callback with the following arguments:

   oRequest

   The same value that was passed in as the first argument to sendRequest

   oParsedResponse

   An object literal containing the following properties:

   oParsedResponse properties

   Property	Type	Description
   tId	Number	Unique transaction ID number.
   results	Array	Schema-parsed data results.
   error	Boolean	Indicates data error.
   cached	Boolean	Indicates cached response.
   meta	Object	Schema-parsed meta data.

   oPayload

   The same value as was passed in to the argument of the oCallback object literal

Instantiating a DataSource

To create a DataSource instance on your page, choose the appropriate DataSource class for the type of data you are using and pass in a reference to the data in the constructor. The simplest subclass available is LocalDataSource, which is meant to support data held in local memory (e.g., a JavaScript array, a JSON struct, a local XML document, a delimited string, or an HTML table element). When data is needs to be accessed from a server, you can instantiate either an XHRDataSource or a ScriptNodeDataSource. A FunctionDataSource allows you to define a function to return data in a highly customizeable manner. As a convenience, the YAHOO.util.DataSource constructor will return the appropriate subclass for you based on the type of live data you pass in. However, if you need a ScriptNodeDataSource, or a LocalDataSource pointing to delimited text, please note that you'll need to specify the dataType in the constructor explicitly.

// DataSources for all kinds of local data
var dsLocalArray = new YAHOO.util.LocalDataSource(["apples", "broccoli", "cherries"]);
var dsLocalJSON = new YAHOO.util.LocalDataSource({
    found: 3,
    total: 20,
    results: [
        {name: "apples", type:"fruit", color: "red"},
        {name: "broccoli", type:"veg", color: "green"},
        {name: "cherries", type:"fruit", color: "red"}
    ]
});
var dsLocalTable = new YAHOO.util.LocalDataSource(YAHOO.util.Dom.get("myTable"));
 
// DataSource for a server accessed over XHR
var dsXHR = new YAHOO.util.XHRDataSource("http://local_path_or_path_to_proxy");
 
// DataSource for a remote location with dynamic script nodes
var dsScriptNode = new YAHOO.util.ScriptNodeDataSource("http://path_to_webservice");
 
// DataSource for a JavaScript function that returns data
var dsFunction = new YAHOO.util.FunctionDataSource(function() {
    return ["apples", "broccoli", "cherries"];
});
 

// DataSources for all kinds of local data
var dsLocalArray = new YAHOO.util.LocalDataSource(["apples", "broccoli", "cherries"]);
var dsLocalJSON = new YAHOO.util.LocalDataSource({
    found: 3,
    total: 20,
    results: [
        {name: "apples", type:"fruit", color: "red"},
        {name: "broccoli", type:"veg", color: "green"},
        {name: "cherries", type:"fruit", color: "red"}
    ]
});
var dsLocalTable = new YAHOO.util.LocalDataSource(YAHOO.util.Dom.get("myTable"));
 
// DataSource for a server accessed over XHR
var dsXHR = new YAHOO.util.XHRDataSource("http://local_path_or_path_to_proxy");
 
// DataSource for a remote location with dynamic script nodes
var dsScriptNode = new YAHOO.util.ScriptNodeDataSource("http://path_to_webservice");
 
// DataSource for a JavaScript function that returns data
var dsFunction = new YAHOO.util.FunctionDataSource(function() {
    return ["apples", "broccoli", "cherries"];
});
 

Setting responseType and responseSchema

DataSource uses the responseType value to determine which parsing algorithm to use, and the responseSchema values determine what data gets parsed out for use by the calling widget. Most DataSources can sniff the responseType value directly from the response, but in some cases you may have to explicitly set the appropriate responseType from the following static constants:

• TYPE_JSARRAY
• TYPE_JSON
• TYPE_XML
• TYPE_TEXT
• TYPE_HTMLTABLE

var xmlDataSource = new YAHOO.util.XHRDataSource("http://myxmlwebservice");
xmlDataSource.responseType = YAHOO.util.XHRDataSource.TYPE_XML;
 

var xmlDataSource = new YAHOO.util.XHRDataSource("http://myxmlwebservice");
xmlDataSource.responseType = YAHOO.util.XHRDataSource.TYPE_XML;
 

Defining a schema for your data allows DataSource to cache and return only the data your widgets will consume, thus minimizing the browser footprint and/or latency of your application. A responseSchema is an object literal of pointers that depend on your data's responseType:

Simple JavaScript Array

The simplest type of DataSource points to an array of strings, such that each result is a string.

Sample data

// A simple array of strings is only considered to contain one field of data
["apples", "broccoli", "cherries"]
 

// A simple array of strings is only considered to contain one field of data
["apples", "broccoli", "cherries"]
 

Sample schema

myDataSource.responseSchema = {
    fields: ["name"]
};
 

myDataSource.responseSchema = {
    fields: ["name"]
};
 

Complex JavaScript Array

A more common type of DataSource points to a complex array of nested arrays or JSON objects that represents a tabular (two-dimensional) data structure. Fields are used to assign a name to each field of data.

Sample data

// Nested arrays - field order matters and must be one-to-one to the data
[
    [ "apples", "fruit", "red" ],
    [ "broccoli", "vegetable", "green" ],
    [ "cherries","fruit", "red" ]
]
myDataSource.responseType = YAHOO.util.XHRDataSource.TYPE_JSARRAY;
myDataSource.responseSchema = {
    fields: [ "name", "type", "color" ]
};
 
// Nested objects - field order doesn't matter and not all data is required to have a field
[
    { type: "fruit", color: "red", name: "apples" },
    { name: "broccoli", color: "green", type: "vegetable" },
    { color: "red", name: "cherries", type: "fruit" }
]
 

// Nested arrays - field order matters and must be one-to-one to the data
[
    [ "apples", "fruit", "red" ],
    [ "broccoli", "vegetable", "green" ],
    [ "cherries","fruit", "red" ]
]
myDataSource.responseType = YAHOO.util.XHRDataSource.TYPE_JSARRAY;
myDataSource.responseSchema = {
    fields: [ "name", "type", "color" ]
};
 
// Nested objects - field order doesn't matter and not all data is required to have a field
[
    { type: "fruit", color: "red", name: "apples" },
    { name: "broccoli", color: "green", type: "vegetable" },
    { color: "red", name: "cherries", type: "fruit" }
]
 

Sample schema

myDataSource.responseSchema = {
    fields: [ "name", "color" ]
};
 

myDataSource.responseSchema = {
    fields: [ "name", "color" ]
};
 

JSON

When a DataSource returns JSON data, fields point to the location of the results using dot-notation. Optionally, metaFields may be defined to access additional meta data. Location pointers to JSON data should be valid JavaScript identifiers, including dot notation or square bracket notation for array indices or quoted strings ("this.is[4]['valid identifier']", but "this-is.not valid"). For convenience, the leading dot can be omitted for dot notation location keys (".foo" is equivalent to "foo" is equivalent to "['foo']"). Use of dot notation is encouraged.

Sample data

{
    "Response" : {
        "Results" : [
            { "id":0, "obj": { "nested":"foo" }, "arr": [null, { "nested 2": 0 } ] },
            { "id":1, "obj": { "nested":"bar" }, "arr": [null, { "nested 2": 3 } ] },
            { "id":2, "obj": { "nested":"baz" }, "arr": [null, { "nested 2": null } ] },
            //...
        ],
        "Total" : 1358,
        "Important" : {
            "to": {
                "me": "pot o' gold"
            }
        }
    },
    "Session" : "12345678"
}
 

{
    "Response" : {
        "Results" : [
            { "id":0, "obj": { "nested":"foo" }, "arr": [null, { "nested 2": 0 } ] },
            { "id":1, "obj": { "nested":"bar" }, "arr": [null, { "nested 2": 3 } ] },
            { "id":2, "obj": { "nested":"baz" }, "arr": [null, { "nested 2": null } ] },
            //...
        ],
        "Total" : 1358,
        "Important" : {
            "to": {
                "me": "pot o' gold"
            }
        }
    },
    "Session" : "12345678"
}
 

Sample schema

myDataSource.responseSchema = {
    resultsList : "Response.Results", // String pointer to result data
    // Field order doesn't matter and not all data is required to have a field
    fields : [
        { key: "id" },                    // simple location
        { key: "obj.nested" },            // dot notation works
        { key: "['arr'][1]['nested 2']" } // bracket notation works
    ],
    metaFields : {
        // oParsedResponse.meta.totalRecords === 1358
        totalRecords : "Response.Total",
 
         // oParsedResponse.meta.something === "pot o' gold"
        something : "Important.to.me"
    }
};
 

myDataSource.responseSchema = {
    resultsList : "Response.Results", // String pointer to result data
    // Field order doesn't matter and not all data is required to have a field
    fields : [
        { key: "id" },                    // simple location
        { key: "obj.nested" },            // dot notation works
        { key: "['arr'][1]['nested 2']" } // bracket notation works
    ],
    metaFields : {
        // oParsedResponse.meta.totalRecords === 1358
        totalRecords : "Response.Total",
 
         // oParsedResponse.meta.something === "pot o' gold"
        something : "Important.to.me"
    }
};
 

XML

A schema for XML data requires a pointer to each result node, in additional to defining fields of data held in each node. Optionally, a metaNode and metaFields may be defined to access additional meta data. Location pointers to XML data can be valid XML node names, attribute names, or preferably, XPath syntax locators. Set the useXPath property to true for XPath support in your locators.

Sample data

<myroot rootatt='5'>
    <top>topvalue</top>
    <second nested='nestedsecond' />
    <allitems>
        <livehere>
            <item type='foo'>
                <name type='nametype0'>Abc</name>
                <rank>0</rank>
                <subitem>
                    <name type='subnametype0'>subABC</name>
                    <age>10</age>
                </subitem>
            </item>
            <item type='bar'>
                <name type='nametype1'>Def</name>
                <rank>1</rank>
                <subitem>
                    <name type='subnametype1'>subDEF</name>
                    <age>11</age>
                </subitem>
            </item>
            </livehere>
        </allitems>
    </myroot>
 

<myroot rootatt='5'>
    <top>topvalue</top>
    <second nested='nestedsecond' />
    <allitems>
        <livehere>
            <item type='foo'>
                <name type='nametype0'>Abc</name>
                <rank>0</rank>
                <subitem>
                    <name type='subnametype0'>subABC</name>
                    <age>10</age>
                </subitem>
            </item>
            <item type='bar'>
                <name type='nametype1'>Def</name>
                <rank>1</rank>
                <subitem>
                    <name type='subnametype1'>subDEF</name>
                    <age>11</age>
                </subitem>
            </item>
            </livehere>
        </allitems>
    </myroot>
 

Sample schema

myDataSource.useXPath = true;
myDataSource.responseSchema = {
    metaFields: {
        rootatt: "/myroot/@rootatt",
        topnode: "//top",
        nestedatt: "//second/@nested"
    },
    resultNode: "item",
    fields: [
        { key: "type", locator: "@type" },
        { key: "rank", parser: "number" },
        "name",
        { key: "subnameatt", locator: "subitem/name/@type"} ,
        { key: "age", locator: "subitem/age", parser: "number" }
    ]
};
 

myDataSource.useXPath = true;
myDataSource.responseSchema = {
    metaFields: {
        rootatt: "/myroot/@rootatt",
        topnode: "//top",
        nestedatt: "//second/@nested"
    },
    resultNode: "item",
    fields: [
        { key: "type", locator: "@type" },
        { key: "rank", parser: "number" },
        "name",
        { key: "subnameatt", locator: "subitem/name/@type"} ,
        { key: "age", locator: "subitem/age", parser: "number" }
    ]
};
 

Delimited Text

When using this relatively simple type of data structure, we must make a small set of assumptions:

• There is a known and constant string delimiter of records.
• Within each record, there is a known and constant string delimiter of data fields that is not equal to the record delimiter.
• There are no escaped delimiter characters in the data

Sample data

apples,fruit,red|broccoli,vegetable,green|cherriesfruit,red
 

apples,fruit,red|broccoli,vegetable,green|cherriesfruit,red
 

Sample schema

myDataSource.responseSchema = {
    fieldDelim : ",",
    recordDelim : "|"
};
 

myDataSource.responseSchema = {
    fieldDelim : ",",
    recordDelim : "|"
};
 

HTML Table

DataSource supports consuming data out of an HTML table element. By default, DataSource will iterate over all TR elements of all TBODY elements of the given TABLE to parse out data. If there are any non-data elements (like summary rows or the message TBODY in DataTable), they should first be removed from the DOM.

Sample data

<-- HTML tables - field order matters -->
<table>
<thead>
    <tr>
        <th>Name</th>
        <th>Type</th>
        <th>Color</th>
    </tr>
</thead>
<tbody>
    <tr>
        <td>Apples</td>
        <td>Fruit</td>
        <td>Red</td>
    </tr>
    <tr>
        <td>Broccoli</td>
        <td>Vegetable</td>
        <td>Green</td>
    </tr>
    <tr>
        <td>Cherries</td>
        <td>Fruit</td>
        <td>Red</td>
    </tr>
</tbody>
</table>
 

<-- HTML tables - field order matters -->
<table>
<thead>
    <tr>
        <th>Name</th>
        <th>Type</th>
        <th>Color</th>
    </tr>
</thead>
<tbody>
    <tr>
        <td>Apples</td>
        <td>Fruit</td>
        <td>Red</td>
    </tr>
    <tr>
        <td>Broccoli</td>
        <td>Vegetable</td>
        <td>Green</td>
    </tr>
    <tr>
        <td>Cherries</td>
        <td>Fruit</td>
        <td>Red</td>
    </tr>
</tbody>
</table>
 

Sample schema

myDataSource.responseSchema = {
    fields: [ "name", "type", "color" ]
};        
 

myDataSource.responseSchema = {
    fields: [ "name", "type", "color" ]
};
 

Type Conversion

Data supplied by the DataSource may need to be type-converted before it can be consumed. This is especially true for XHRDataSources or progressive enhancement scenarios where data may be intended to be of type Number, but gets loaded as type String. Parser functions can be assigned to convert data on a per field basis.

Built-in parsers include parseDate which will handle any date that the native JavaScript Date.parse() method handles (notice not all date formats are supported) and parseNumber which will try to parse a number whether an integer or float. A custom function may also be defined by the implementer to override these built-in functions.

As of the 2.6.0 release, the following string shortcuts may be used to point to the built-in parser functions:

• "date" points to YAHOO.util.DataSource.parseDate
• "number" points to YAHOO.util.DataSource.parseNumber
• "string" points to YAHOO.util.DataSource.parseString

Sample data

{
    "Inventory" : {
        "Produce" : [
            { "name": "apples", type: "fruit", color: "red", quantity: "50" },
            { "name": "broccoli", type: "fruit", color: "green", quantity: "35" },
            { "name": "cherries", type: "fruit", color: "red", quantity: "100" },
        ],
        "Total" : 20
    }
}
 

{
    "Inventory" : {
        "Produce" : [
            { "name": "apples", type: "fruit", color: "red", quantity: "50" },
            { "name": "broccoli", type: "fruit", color: "green", quantity: "35" },
            { "name": "cherries", type: "fruit", color: "red", quantity: "100" },
        ],
        "Total" : 20
    }
}
 

Sample schema

myDataSource.responseSchema = {
    resultsList : "Inventory.Produce",
    fields : [
        "name",
        "type",
        // This field will convert incoming data using a custom function 
        {key: "color", parser: myColorToRGB },
        // This field will convert incoming data to type Number
        {key: "quantity", parser: "number" }
    ],
    metaFields : {
        // oParsedResponse.meta.totalRecords === 1358
        totalRecords : "Response.Total",
 
         // oParsedResponse.meta.something === "pot o' gold"
        something : "Important.to.me"
    }
};
 

myDataSource.responseSchema = {
    resultsList : "Inventory.Produce",
    fields : [
        "name",
        "type",
        // This field will convert incoming data using a custom function 
        {key: "color", parser: myColorToRGB },
        // This field will convert incoming data to type Number
        {key: "quantity", parser: "number" }
    ],
    metaFields : {
        // oParsedResponse.meta.totalRecords === 1358
        totalRecords : "Response.Total",
 
         // oParsedResponse.meta.something === "pot o' gold"
        something : "Important.to.me"
    }
};
 

Caching

The DataSource component can cache data in a local JavaScript array, which is especially helpful for reducing network traffic to remote servers. You can set the maxCacheEntries property to the number of responses you want held in the cache. When this number is exceeded, the oldest cached response will be dropped. The data in the cache is already fully processed, extracted, parsed and ready to be returned to the caller. On a cache hit, those events or overridable methods meant to signal a connection to the live data or handling of raw responses will not be fired or called.

myDataSource.maxCacheEntries = 4;
 

myDataSource.maxCacheEntries = 4;
 

The cache will be updated when polling but won't be used for retrieval.

Cache entries are indexed by the oRequest (first) argument of sendRequest. There will be one entry per request up to the total of maxCachedEntries.

Polling

The setInterval method can be used to repeatedly poll for data. The first argument is the number of milliseconds to wait between requests, and the following three arguments are the same as those of sendRequest. This method returns a numeric identifier that can be used to cancel the polling.

Polling can be cancelled by calling clearInterval and passing it the identifier returned by the call to setInterval. All polling can be cancelled by calling clearAllIntervals. Please note that existing transactions will not be canceled.

Custom Events and "doBefore" Abstract Methods

DataSource offers several hooks can be used to monitor the progress of the data request/response cycle or perform transformations on the response along its journey back to the callback.

cacheRequestEvent

Fired when a request is made to the local cache.

cacheResponseEvent

Fired when data is retrieved from the local cache.

requestEvent

Fired when a request is sent to the live data source.

responseEvent

Fired when live data source sends response.

doBeforeParseData

This overridable abstract method gives implementers an opportunity to munge the data before it is schema-parsed. Implementers should be sure to return data in a ready-to-parse state to avoid errors.

responseParseEvent

Fired when response is parsed.

dataErrorEvent

Fired when a data error is encountered.

doBeforeCallback

This overridable abstract method gives implementers an opportunity to access the data before it has been cached or returned to the callback. Implementers should be sure to return data in a ready-to-return state to avoid errors.

Using XHRDataSource

The XHRDataSource class uses the YUI Connection Manager to retrieve data from a server. Implementers should be aware of browser security restrictions that prevent XHR connections across domaina. The following properties are available for configuration:

Special considerations need to be made when using a DataSource that supports asynchronous request/response cycles, including state validation in the sendRequest() callback's success and failure handlers. In the case of YUI widgets consuming DataSource, these handlers will want to verify that the widget instances themselves are still valid.

Using ScriptNodeDataSource

The ScriptNodeDataSource class uses the YUI Get Utility to retrieve data from a remote server without the need for a proxy. Note: ScriptNodeDataSource requires that the webservice facilitate a callback mechanism in its response. The following properties are available for configuration:

Special considerations need to be made when using a DataSource that supports asynchronous request/response cycles, including state validation in the sendRequest() callback's success and failure handlers. In the case of YUI widgets consuming DataSource, these handlers will want to verify that the widget instances themselves are still valid.

YUI on Mobile: Using DataSource Utility with "A-Grade" Mobile Browsers

The core functionality of the DataSource utility operates without any major issues on high-end mobile platforms, keeping in mind the following preliminary list of smart phone known issues:

• XML is not currently supported on Android devices.

Support & Community

The YUI Library and related topics are discussed on the on the YUILibrary.com forums.

Also be sure to check out YUIBlog for updates and articles about the YUI Library written by the library's developers.

Filing Bugs & Feature Requests

The YUI Library's public bug tracking and feature request repositories are located on the YUILibrary.com site. Before filing new feature requests or bug reports, please review our reporting guidelines.