Ipkg API

Overview

Currently Preware populates it's data by pulling the entire packages index from each feed and doing a slow parsing of the text files. This is, as mentioned, slow, as well as wasteful since the majority of the data in the Packages index is already known (anything that wasn't updated since the last feed update). What is proposed is an API to help deal with these limitations. This API spec will be in flux as it is discussed.

Requirements

There are a few requirements that this API should be able to meet:

• It must accept and return properly formatted JSON
• It must be capable of giving only information after a provided time index
• It must return the time index of when the result set was queried
• It must allow for arbitrary queries based on any combination of data available
• It must allow for limiting the amount of data returned (only version numbers, only appid's etc)
• It must return enough data that every package returned can be uniquely identified (appid, architecture, feed, version)
• It must support storage and query of every key within the ipkg packages file (including the ones jammed into the source line
• (Optional)Support for predefined queries
• (Optional)Support for dynamic storage of arbitrary new keys if a new one should crop up

Prototype description

• All API requests will include a serial (seconds since epoch) and only results that were created or modified after that serial will be returned.
• You will be able to specify a set of Filters as key/value pairs, only results matching these filters will be returned.
• You will be able to specify what data you are requesting and only that data (plus anything else required to uniquely identify the package) will be returned.
• If you omit the array of requested data, all data for the matching packages will be returned.

Prototype demo

Here is a demo query requesting the version number and description of armv7 preware newer than 1234567890 seconds since epoch Please note that this would return results that match the appid and arch from any feed

{"Serial":"1234567890","Filter":{"appid":"org.webosinternals.preware","arch":"armv7"},"Request":["version","description"]}

Here is a demo response that would be returned, notice that the Filters are returned as well. At a minimum, all package responses will include the appid, feed, arch and download URL (URL encoded) A new serial is returned that corresponds with the time since epoch that this data was pulled

{"Serial":"2244668800","Response":[{"appid":"org.webosinternals.org","feed":"webos-internals","arch":"armv7","version":"1.7.0","description":"Prewares package description (URL encoded)"},{"appid":"org.webosinternals.org","feed":"webos-internals-testing","arch":"armv7","version":"1.7.2","description":"Prewares package description (URL encoded)"}]}

A serial will be required with ALL API requests, if you want a full data set, just use a serial of 0 Failure to include a serial will result in no response from the API server

Using the serials, you can get partial responses for updates, if you already know that your package data for the webos-internals armv7 feed is up-to-date as of serial 55667788 you can query for updates to it

{"Serial":"55667788","Filter":{"feed":"webos-internals","arch":"armv7"}}

The return would be very similar to the one shown above with the exception that it would have all packages updated after the serial that is provided, limited to packages in the webos-internals armv7 feed... and most importantly... Because you did not provide a Request stating what pieces of data you want, you get everything the API server has about those packages, this is most likely the way that Preware would pull it's updates

A query with multiple options for a filter can also be performed. The first example would return packages from all feeds but if you wanted results from only the webos-internals and webos-patches feeds that is accomplished by supplying an array as the feed filter as in the following example

{"Serial":"1234567890","Filter":{"appid":"org.webosinternals.preware","arch":"armv7","feed":["webos-internals","webos-patches"]},"Request":["version","description"]}

Predefined Queries

These would be accomplished by sending a query similar to the following:

{"Serial":"1234567890","Filter":{"appid":"org.webosinternals.preware","arch":"armv7"},"Request":["preset":"preware-feed-pull"]}

The API server would be pre-programmed for certain predefined queries, it would know what data should be returned as part of that preset. We could make as many of these as needed, but I suspect that only a small handfull would be needed. This would aid code simplicity when it comes time to implement this API on the client side as well as making the client side code easier to maintain.

Should we need some new field sent along for some new feature in a package that uses the API, we could adjust the response in one place, on the server, and that newly required data would be sent along.

Extra Considerations

Source Line Split

Right now, we are jamming a lot of "extra" information into the Source: line in the Packages index as a JSON object. With this API, it would be fairly trivial to split these out into separate references. For example, here is the Packages index information for Preware v1.7.0

Package: org.webosinternals.preware
Version: 1.7.0
Section: System Utilities
Architecture: arm
Maintainer: WebOS Internals <support@webos-internals.org>
MD5Sum: 6e896b02fa512c0e811b352b4fa775e6
Size: 334828
Filename: org.webosinternals.preware_1.7.0_arm.ipk
Source: { "Source":"git://git.webos-internals.org/applications/preware.git", "Feed":"WebOS Internals", "Type":"Application", "Category":"System Utilities", "LastUpdated":"1309401687", "Title":"Preware", "FullDescription":"The revolutionary webOS on-device installer capable of installing advanced homebrew applications, themes, patches, kernels, services and plugins. Preware is also able to access official Palm App Catalog, Web Distribution and Beta Distribution feeds.<br><b>Requires webOS 1.4.0 or later</b>.<br>Changelog:<br>1.7.0: Rolled up all the beta release features for a TouchPad-compatible public release.<br> 1.6.8: Enabled back tap on header for all devices.<br> 1.6.7: Fixed full-screen formatting of ipkg log screen.<br> 1.6.6: Preware is now fully compatible with the TouchPad, using the full screen area.<br> 1.6.5: Now useable on devices without a back gesture. Set the wget user-agent to Preware when retrieving indexes and packages.<br> 1.6.4: Fixed a problem when loading the saved packages list.<br> 1.6.3: WebOS Internals has been formally notified by Richard Kerris (HP Vice President of Worldwide Developer Relations) that open access to the webOS app catalog feed (including the beta and web feeds) is no longer available to third party developers or applications. Please direct any questions about this decision (which is final and permanent) to HP, not to WebOS Internals. Removed all HP webOS app catalog feeds from new installations of Preware.<br> 1.6.2: Added the clock themes feed (disabled by default).<br> 1.6.0: webOS 2.x Just Type integration. Fixed package list ordering bug. Layout fixes for Pixi/Veer.<br> 1.5.9: Fixed parse errors resulting from packages with very large descriptions. Fixed problems with updates to app catalog apps.<br> 1.5.8: Moved blacklisting to feed load to speed up listing. Suggests a reload after changing the blacklist. Fixed homebrew feed/catalog feed same appid bug.<br> 1.5.7: Added check to make sure package has a title before trying to search it.<br> 1.5.6: Removed support for Visibility feed value - use a category filter to exclude Unavailable packages if desired.<br> 1.5.5: Added support for Visibility feed value. Fix display of screenshots from devices with a different resolution. Test fix for custom feed on network with no internet connection. Style updates for pre3 screen size.<br> 1.5.4: Updated French translations (courtesy of Clément).<br> 1.5.3: Reinstated banner notification of manual saved package list update.<br> 1.5.2: Check if package dates are actually dates. Removed banner notification of saved package list being saved.<br> 1.5.1: Added support for optware testing feeds.<br> 1.5.0: You must upgrade to this version for full webOS 2.0 compatibility.<br> 1.4.9: Better compatibility with unknown future webOS versions.<br> 1.4.8: Now calls the correct software manager updates screen on webOS 2.0.<br> 1.4.7: Fixed update, install and remove status messages on webOS 2.0.<br> 1.4.6: New App Catalog theme preferences thanks to Garrett92C. Added support for MaxWebOSVersion in the feeds.<br> 1.4.5: Added support for identifying when running on a Pre 2 device.<br> 1.4.4: Updated the version check to support future webOS versions.<br> 1.4.2: Added Category as a blacklist option.<br> 1.4.1: Added a Changelog button to the Help scene.<br> 1.4.0: Added help to preferences scene. Fixed swipe-to-delete bug in blacklist. Updated Italian translations (courtesy of Darkmagister). Added Get Info button to package install scene. Added Send button to ipkg log scene. Fixed bug when loading large package descriptions.<br> 1.3.8: Updated French translations (courtesy of Yannick LE NY).<br> 1.3.6: Preware now supports installation of application package files (.ipk files) directly from URLs, email attachments, and local files. Also supported are Send to Preware links on homebrew application gallery and news websites (this requires the Neato! application to also be installed). If Preware continually asks you about file associations on every single launch, even after you have answered Yes to the questions it asks, then you may have a damaged file association table and will need to run the Emergency MimeTable Reset tool and then reboot to repair it.", "Homepage":"http://www.webos-internals.org/wiki/Application:Preware", "Icon":"http://www.webos-internals.org/images/e/e4/Icon_Preware.png", "Screenshots":[ "http://www.webos-internals.org/images/3/36/Preware_ss1.png", "http://www.webos-internals.org/images/d/d5/Preware_ss1a.png", "http://www.webos-internals.org/images/9/97/Preware_ss2.png", "http://www.webos-internals.org/images/6/6f/Preware_ss2a.png", "http://www.webos-internals.org/images/e/e6/Preware_ss2b.png", "http://www.webos-internals.org/images/8/8a/Preware_ss3.png", "http://www.webos-internals.org/images/a/a1/Preware_ss4.png", "http://www.webos-internals.org/images/2/28/Preware_ss5.png", "http://www.webos-internals.org/images/e/e1/Preware_ss6.png", "http://www.webos-internals.org/images/5/55/Preware_ss7.png", "http://www.webos-internals.org/images/9/94/Preware_ss8.png", "http://www.webos-internals.org/images/e/ea/Preware_ss9.png", "http://www.webos-internals.org/images/b/bc/Preware_ss10.png", "http://www.webos-internals.org/images/7/75/Preware_ss11.png" ], "License":"GPL v2 Open Source", "MinWebOSVersion":"1.4.0" }
Description: Preware

notice the long and nasty Source: line... we can't change this in the actual Packages index file because it would break the spec for an ipkg feed, but we can have the API server do a little processing to make all of these able to be requested individually.

API server updates

We can either have the API servers run a database update on a schedule, or (and I like this option better), it would be pretty easy to have the autobuilder capable of notifying the API server (possibly servers) of when it runs a pass where a new package was built. The flow would be about like this.

Cron triggers Autobuilder -> Autobuilder runs -> No Packages build -> Time passes -> Cron triggers Autobuilder -> At least 1 package built -> Autobuilder triggers API server -> API server updates -> Time passes...

and so on. This second option would be fairly trivial to implement since an update of the API server can be triggered through a simple GET request and we can limit the IP's allowed to make a GET to that resource so that only the autobuilders (and localhost) can trigger an update.