Object Service Architecture

Web Annotation Service

  September 15, 1998  

Executive Summary

Military intelligence analysis and command and control organizations as well as industrial and educational institutions increasingly use the World Wide Web to share and make available information about their "organizational memory" or the situations they monitor.  Today's Web is a global document repository where authors actively publish information and browsers passively view it.   The objective of the OSA/Annotations project is to augment the Web with the basic capability for third party viewers to augment or annotate content supplied by authors, with or without author knowledge.  This can allow closed groups of collaborators or large "communities of interest" to actively add expertise to information posted on the Web so that the community information base can rapidly and incrementally improve based on the expertise of all its members, not just primary authors.  Our approach is to develop an open Annotation service that can be used in conjunction with standard Web browsers.  The service is invoked by a general purpose URL interceptor which installs the side-effect (transducer or extension) of searching for document annotations as the document itself is retrieved.  The basic annotation service can be tailored in a number of ways, for instance, to only display annotations from certain groups or of certain types.  Higher-level facilities like pedigree drill down and filtering, argumentation, or workflow can be built using the OSA/annotations service as a primitive.

Problem Statement

In the JTF/ATD Web Server project, DARPA is using an idiosyncratic object-oriented "web" to represent and monitor situation description and assessment information about evolving command and control events.  Though it is much less structured at present, information authors use the World Wide Web for much the same purpose, to post information content about topics of interest within private Intranets or to the public Internet.  To date, Webs are populated by authors who are authorized to add content.  Surfers who access the information provided on Web pages are generally passive viewers, with no direct way to add their expertise to the subject matter.  Yet these same viewers have an interest in the subject matter they are accessing and may have additional knowledge (more recent, deeper, better analyzed, corrections, viewpoints, clarifications, ratings, filters) that could be shared across a community to augment published content.  For example, a user of a global map facility could use a general annotation facility to report new roads in a new subdivision or inaccurate addresses; a soldier in the field could report that a situation has changed, for instance, a bridge is temporarily disabled.  If system applications like DBMS, workflow, or collaborative authoring are open for extensional services like annotations, then soldiers might use publicly available information, annotating it to adapt it to military use, instead of having the military invest in its own tool sets to get the extensional capabilities it needs.

Objectives

The technology objective of the OSA/Annotation project was to develop an open, scalable annotation service that can be tailored to accomodate a range of custom uses.  The Annotation Service allows existing web content to be augmented or filtered using third-party information, which can be much richer in structure and content. Such information adds value by elaborating on, restricting, providing a background or rationale to, branding, and personalizing the web content based on user metadata.

The infrastructure objective was to understand how to add new services to standard web engines in the same way that we can for ORBs.

The technology transfer objective was to complete a prototype toolkit available to the DARPA AITS application community, the Web vendor and user communities, and the Web standards community W3C.

Architecture

The architectural elements of a basic Annotation Service include:

• annotations and annotation sets - annotations consist of source object and anchor, annotation body, and possibly other metadata.  An annotation set is a collection of annotations on a given object.
• annotation authoring - a person or program creates an annotation set and stores it in an annotation repository.
• annotation repository - stores and retrieves annotations given a target URL and other retrieval criteria
• URL interceptor - captures URL at dispatch
• annotation service extension - the core service that wraps the normal URL access with side-effects to access the annotation repository and compose the returned document and annotations on it.
• annotation composition - composes the returned document with its annotations
• annotation viewing - simultaneous presentation of returned document and annotations using some style
• annotation service management interface - an API and GUI for controlling annotations (turn on, off, select options and filters)

Design Space An open Annotation service involves a rich design space.  A design space describes, for each part of the architecture, the range of alternative design choices that can be supported.  The design space can be viewed as a formal AND/OR graph structure in which AND layers describe required components and OR layers describe tailoring alternatives.  The design space enumerates the applicable alternatives and tradeoffs, can identify the basic architectural structure of the service, can support tailoring of the service, and can be used to generate specifications. We found little prior work describing the design space for Annotations, and wanted to explore alternatives that seemed promising but had not yet been tried. For our purposes, we describe the design space informally as a hierarchy of objectives and requirements.

The following is a list of the issues and alternatives we encountered in reviewing prior work and in designing our prototypes:

• annotation representation -- annotations represented as annotation objects: <URL, anchor, type, annotation body, metadata>
• • annotation bodies can consist of text, code, XML, HTML, or other formats. Some formats such as HTML can provide implicit support for graphics and sound.
  • anchor points in source or target documents can be specified by URL, tag, offset, pattern, query, or other means. A library for finding anchor points wold be useful for applications other than just annotations.
  • new kinds of annotation metadata can be specified for different purposes:  author, timestamp, type, default colors, pedigree and provenance, security, and so on.
  • annotations can have behavior, e.g. accumulating statistics or votes over time
  • human readable format - for viewing saved annotations available in the repository or via the viewer
  • recursion: annotations, themselves objects or MIME types, might be annotated
  • annotations can be used for system reasons
  • • to mark sections of a document "Secret" and hide content
    • typed annotations with MIME types, e.g. voice.
  • annotations can be used to form relationships between a source object and a target object.  The relationships can have properties.
  • • annotations may be used as views to support transclusion, which is the virtual copying and possible caching of content while maintaining derivation to create a relationship between two objects.
    • • there may be a set of presentation styles to designate quoted material
      • annotations should obey transcopyright rules
  • Intermediary function: annotation selection requires monitoring which URLs the browser fetches
  • • also called an interceptor or transducer
    • may be implemented as a proxy intermediary on server, client, or in between that simply captures URL refs
    • may implemented as a browser extension - however, browsers provide different extension mechanisms, requiring different intermediary implementations. Also, the browser may not support URL interception in its extension mechanisms, e.g. Netscape does not allow plugins to register for handling HTML types.
    • may be implemented as a Java applet calling native code to interface with the browser
    • may be implemented as JavaScript code between sibling frames, combined with server intermediary to localize and de-localize all references on page. Applets and plugins embedded in a page may have problems accessing data from their server this way.
    • current solutions are unsatisfactory in that they compromise security or require extra-browser configuration.
  • The side-effect action of the intermediary typically involves retrieving annotations associated with the URL.  In the OSA/Network-Performance-Monitor-Service project, the intermediary can be used to launch a companion performance monitor to track and report back on network weather.  Many extensions are enabled by the same intermediary mechanism (security, advertising, advertising suppressor, filtering, counting, ...).
  • The intermediary/interceptor is analogous to the interceptor or portable object adapter dispatch traps in a CORBA system, so this is like a URL Request Broker and the web architecture augmented with extensions mechanisms is like the OMG architecture.  We need to merge the Web URL Request Broker and CORBA so services on the ORB can operate on data on the web, e.g., so ORB versioning can guard the web document model.
  • tradeoffs exist regarding placement of the intermediary/interceptor on a server, on the client, or in between: client placement provides better performance but requires configuration on the client. Server placement adds an extra retrieval hop, increasing latency for page retrieval.
• the annotation repository
• • can be a file, DBMS, search engine, or other get-put associative retrieval engine
  • may use a parseable format such as XML or HTML so arbitrary users encountering the repository can see some representation of its content.
  • may publish metadata about itself to search engines so it can be located
  • can be local to a user, shared by a work group, public, or federated and accessible to a "community of interest", leading to individual, group, or public annotations.
  • for scaling to large numbers of users, the repository needs a replicated data store similar to Usenet newsgroups, or may perhaps use newsgroups directly as a file system, if security is no concern.
• presentation
• • viewing of annotation sets can vary according to styles supported in a browser or editor and user preference.  The style of presentation can vary: in-line annotations, post-it notes, annotations juxtaposed with content in separate frames, the equivalent of page margins where only the annotations relevant to a page are shown in the juxtaposed frame etc. Multimedia voice or animation might also be played during the presentation.
  • annotations need not be embedded in the content as presented, thus allowing opaque information (e.g images that cannot be analyzed) to be annotated with text and vice-versa.
  • presentation styles can be "active" - i.e., include behavior that affects other parts of user environment, such as accepting user input for ratings
  • annotation-based metadata can be used to create and control views
• scalability of delivery mechanism to support large numbers of clients and annotations and avoid information overload
• • support discovery of communities of interest from groups too large to know one another: for example
  • • support messaging to all authors of annotations in a particular thread.
    • allow filtering by list of authors so only annotations by members of the list are viewed
  • browser and annotation viewer should be able to retrieve and present content in parallel on client
  • support community rating of annotation content and filtering by rating on retrieval
  • support expiration dates - old annotations are automatically deleted after a time
  • allow expiration to be overruled, e.g. annotations with high ratings live longer
  • annotations should be kept small and delivered via lightweight mechanisms e.g. IP datagrams
• annotation control
• • user-specified filters or queries can permissively select subsets of annotations for viewing.
  • Security filters can restrict some viewers from viewing some annotations
  • filtering annotations by various criteria e.g. author, annotations after 8/3/97, ...
  • active composition: have the server retrieve content from a newsgroup etc. to form the annotation text.
  • delivery time of annotations can be synchronous or asynchronous with delivery of content
  • • provide efficient and low-cost delivery mechanisms to enable quick delivery of many notes
    • "Pulling" information from foreign (e.g. e-mail and news) repositories and presenting it as annotations allows us to treat traditional asynchronous interaction mechanisms as annotation authoring environments.
    • simultaneously arriving annotations from different sources might be viewed as a kind of chat about the document being inspected
    • annotation delivery may be adaptive to user portals and preferences.
  • scale up to support large "hot-spot" documents with many annotations
• use of standard infrastructure
• • provide variety of architectural mechanisms as components to intercept and annotate web content that can be used independently or composed to address different scalability concerns
  • creation of an annotation set can be manual but more commonly will be supported by an authoring environment used to create annotations.  This may involve extending a browser or editor or, for a heterogeneous domain, multiple browsers and editors.
  • annotation authoring and viewing can be non-invasive; i.e. use standard web browsers , requiring little or no modification to current web infrastructure or content
  • static vs. dynamic architectures for annotation delivery.
  • • Static architectures (whereby the system components are pre-wired), are further divided into transparent and non-transparent architectures depending on whether they employ a different URL space for annotation:
    • • Previous efforts have focussed mainly on transparent architectures where annotated content has the same URL as unannotated content.
      • Non-transparent architectures, which map the annotated content to a new URL, can provide a different set of scalability benefits.
    • Dynamic architectures can provide support for large communities of communities by dynamically locating and binding to repositories. This requires supporting metasearch, where annotations are composed with search and metadata services to match  the annotation consumer with the appropriate annotation repositories, using a mechanism similar to that provided by the OSA/Trader.
  • accommodate wide diversity in information sources
  • • interoperate with widely used legacy mechanisms of information authoring by generating wrappers for foreign repositories such as e-mail and news.
    • automate the generation of such repository wrappers using meta-data that characterizes the repository. An example of such wrapping is (semi)automatically figuring out the target and anchor points of an e-mail message being processed as an annotation.
• making the overall architecture extensible and customizable
• • The composition and presentation interfaces of OSA/annotations are open and dynamically customizable. This allows new classes to be written and added at runtime to support different composition or presentation styles. These classes can be "active" and can incorporate information sources of arbitrary complexity.
  • higher-level facilities like pedigree drill down, argumentation, or workflow can use the OSA/annotations service as a primitive service.
  • • workflow systems typically support some kind of ability to create relationships among objects governed by different tools.
    • pedigree (e.g. drill-down) visualization mechanisms and pedigree-based information filters.
    • • • Pedigree information could be a query on dynamically changing information sources
        • link mechanism that is used for annotation argumentation will be adapted to support static and dynamic (query-based) specification of the annotation pedigree
        • information source certainty and believability metadata can be accessed
  • support adding "active" composition, presentation behaviors, new anchor types dynamically
  • call back mechanism - this mechanism allows a document provider to associate with web pages she authors a list of actions that can be taken with respect to the page.  One such action is to associate with the page a list of annotation repositories to associate with the page.  These would be searched if the viewer requests access to the page.  This is a server side mechanism.

Limitations of Current Work

At present, browsers have no general mechanism for third parties to provide value-added information to the Web and make it available to others. The state of practice mechanisms available today requires sending email to Web page authors suggesting adding updates or doing this more publicly if the page author includes comment boxes on his web page, which is a very limited form of annotation.

Recent work on Web annotations includes the CERN annotator,  Stanford ComMentor, CritLink, CoNote , Strand/GrAnT and the Net Notions product. While the architectures of some of these systems are open in one way or another (for instance, including the idea of public and group annotations), the implementations tend to require modified web infrastructures, some systems are slow, and none has been widely deployed. CoNotes requires document authors to anticipate places where readers might wish to add commentary or questions by inserting "annotation points" into the original document. CoNotes can only annotate documents under its server's control, and does not support annotating arbitrary web content. In HyperNews and CritLink, the request for annotated content is syntactically different from a request for the unannotated document. HyperNews system is primarily a discussion group tool, but has supported annotation in conjunction with the use of modified browsers (Mosaic and HotJava) in the past. CritLink uses a specialized server called the mediator. The Mediator retrieves a requested page, prefixes all references in it with its Internet domain, and returns the annotated, modified page to the user. All further navigation from links within the returned pages are forced to be processed by the Mediator. This approach requires users to navigate from within the CritLink-processed page instead of by using the browser's bookmarks or direct URL entry. Both NCSA's Mosaic project and ComMentor modify web browsers to augment them with annotation capabilities. The NCSA Mosaic browser supported private annotations and group shared annotations in several releases. Both Strand/GrAnT and the Net Notions product introduce web annotation function without modifying web content, browsers or servers. They are similar in philosophy to the systems that we have built. Strand uses a client intermediary for page interception. The Strand authors noted some problems with changing page layout when inserting annotation indicators and functions, and envisioned that much of the GrAnT functionality might be moved into a Java applet, the approach that JotBot explores. Net Notions is a commercial product from Sideware Systems, Inc. that allows users to affix text annotations "over" arbitrary web pages. Net Notions uses a client process that apparently records current URL and position information when annotations are created.

Other topics related to annotation include:

• transclusion and transcopyright -- the Xanadu notions of virtual inclusion, "which permits composite documents to be constructed by reference to the original components rather than by copying" including permissions for such references.
• link services -- these services explicitly remove references from the original source document and store them separately so that URLs are not embedded within documents.  This allows the same document to be annotated bidirectionally with different collections of links for different purposes.
• relationships -- annotations can be viewed as a kind of relationship.  OMG has defined a Relationship Service for connecting distributed CORBA objects.  The OMG Telecom Task Force has issued an RFP for a Topology Service for managing graphs of relationships (see HP initial submission).
• rating services - PICS and now PICS-NG respectively support the addition of metadata for rating purposes and for more general purposes.  These are related to annotations in that annotations can themselves be viewed as metadata, annotations can use metadata schemes, and conceptually, URL access is overloaded with an additional behavior, to filter or negotiate changes to content to match viewer attributes so a similar interceptor scheme can be used to install PICS into a browser.
• interceptors -- OMG has defined two mechanisms for dynamic dispatch to over load the normal CORBA IOR access mechanism, one is the interceptor from the Security specification (Part I and Part II), the other the Portable Object Adapter (Part I and Part II).  Jigsaw servers support server-side before and after interceptors.

Results

We have constructed two prototype OSA/annotation implementations to illustrate these different aspects of an open annotation service. Early releases of both prototypes were demonstrated at the DARPA PI meeting in San Diego during 10/97. A reference model for annotation architectures is presented in a paper to be published in the HICSS '99 conference. The Server-side prototype modifies the Jigsaw web server and maps annotated URLs to a new URL space. The JotBot prototype takes a client-side approach by using a Java applet to retrieve and compose annotations within the browser, allowing the annotation server to be thin.

The Server-side Prototype supports:

• an Annotation class supported by the annotation repository (implemented using a Java persistent object store)
• • URL descriptor objects - these are container objects that contain annotations for a particular URL
  • annotations
  • • annotation bodies -- can be type text or HTML.
    • oid -- a system-assigned oid for each annotation. Used in link modeling.
    • anchor types supported -- The anchor is either URL or positional. Positional tags are used for embedded annotations.
    • metadata -- currently a timestamp field is supported as an example of metadata.
  • Links that allow annotations on annotations
  • • Link objects are a triplet (source annotation, target annotation, link type)
    • The Link data structure can accommodate multiple link types, and many-to-many links.
    • Link navigation methods are supported.
    • Currently, links are scoped to annotations on the same URL
• Annotation repository wrappers - The annotation repository has three wrappers that support the same annotation API with different benefits:
• • The Java wrapper returns query results as Java data structures. This wrapper can be used by clients that are co-located with the repository
  • The HTTP wrapper supports queries specified as URL fetches (like CGI). It returns query results as human viewable HTML pages. This wrapper can be used by clients anywhere to query the annotation repository
• annotation creation -- via an HTML Form-based interface
• annotation access -- overloads the Get operation in a Jigsaw server - requires the user to use a different URL for annotations than for the original document - The prototype uses jigsaw facilities to associate the two URLs.
• two presentation styles
• • annotations juxtaposed with content in separate frames
  • in-lining annotations at structural or positional anchor points. Structural anchor points are demonstrated by the use of existing HTML tags as an anchor for in-line annotations.
• OSA/Annotations supports audio annotations as is:
• • HTTP servers already have the ability to publish these files as URLs which are replayed by client-side helpers/plug-ins in the usual way.
  • A hyperlink to audio/video shows up embedded in the doc in the usual way, and the helper tool plays the audio when the hyperlink is clicked.

The JotBot prototype supports:

• annotation representation
• • annotation bodies -- Java objects, which could be built dynamically from arbitrary other types, e.g. ontologies, at retrieval time. The code for the "Compose" and "Present" methods is loaded dynamically.
  • anchor types -- keyword for text match within the annotated document
  • metadata --
  • • author
    • creation/expiration dates
    • security class
    • Unidirectional link to a URL or image (meaning determined by loaded behavior)
    • name of Presenter class
    • name of Composer class
    • color to be used by presenter
• annotation creation
• • simple editor applet - working and integrated with viewer. Currently, the editor occupies one browser window, while opening a second browser window for navigational use.
  • editor is a Java V1.0 applet that runs in a browser.
  • the editor controls annotation size (so they can be delivered as datagrams) and does simple content validation
• annotation access -- URL interceptor implemented using a Java native-code library
• • this is now working for demonstration purposes for Netscape only, although the library supports IE as well and the browser restriction can be removed.
• annotation repository --
• • multi-threaded server, can be evolved to use datagrams instead of maintaining IP connections
  • lightweight - process keeps no connection state so it can service more requests
  • supports small annotations, sent asynchronously with respect to retrieval of target page
  • annotations have a default lifetime after which they are deleted, but accept community content ratings. High ratings extend the life of an annotation.
  • allows filtering of annotations by security clearance and author (demoed in release 1, 10/97, later disabled)
  • can also support delivery of larger annotations via an IP socket if this quality of service is needed
  • currently implemented on file but could be migrated to use DB or other store.
• messages to discovered author community
• • send SMTP email to list of annotation authors including reference to annotated page
• dynamically loaded presentation styles
• • several Presentation behaviors were implemented under the same interface
  • • text squares
    • LinkTo - presented as button that causes main browser to display new page of specified URL, in effect allowing new HREFs to be added as annotations to existing pages.
    • audio note that plays .wav file on button press (demoed in release1 10/97, later disabled)
    • image note - displays a .jpg image (demoed in release1 10/97, later disabled)
    • Survey - has behavior/GUI to accept votes pro/con for user-supplied questions
    • Rating - accepts user votes to prolong the life of a given annotation
    • viewer - displays URL content and annotations

We can author annotations without requiring any change to web infrastructure since the editor and server are independent components. We can retrieve and present annotations also non-invasively since the Viewer applet is an independent component. The viewer applet needs only to know when the browser fetches a new URL, and it has an open interface for a browser-dependent interceptor to use. The Survey, Rating, and Messaging functions are innovative in that they adopt features found in other collaboration research for use in an annotations context.

Lessons Learned

It is a restriction of current browser implementations that they don't publish URL fetches, and getting around that restriction - even to a limited extent - has proved difficult. By introducing a client intermediary we can obtain most of the information we need and modify content sent to the browser arbitrarily without being limited by browser security. We should not need to modify infrastructure to get URL info if the browser allows access to it in a way that supports its security controls. It would be desirable for browsers to provide more open interfaces in the future.

During late 1997 and 1998, XML became a hot topic and gained significant momentum as a future web technology that will ultimately complement - but not replace - HTML. XML's support for semantic tagging makes it possible to express annotation objects and links as XML elements. The use of XML could potentially make most web documents accept annotation, and facilitate development of more powerful forms of annotation. An XML-based annotation repository could offer a lighter weight alternative to other data store choices. However, a number of concepts in XML such as semantic tagging and the ability to create external hyperlinks between documents could change the architecture of browsers, and therefore the architecture of annotation systems. The XML developments occurred too late in this contract to really allow time to redirect our work to focus on XML. At present, standard browsers do not yet support XML and the nature of this support remains to be seen. HTML is likely to constitute the vast majority of web documents for some time to come. Given our goal of supporting commonly available technology unobtrusively, we decided to continue to work with existing browser technology.

During the spring of 1998 we worked closely with a local entrepreneur who wanted to license the annotation technology for use on a future commercial web site, as a means to conduct online focus groups and reviews of business plans. This effort gave us some positive feedback about usefulness and usability of the tool as well as providing insight to the work that would be required to deploy the software for commercial use, testing on all browsers and server platforms.

Also notable was the release of the Net Notions product by SideWare, inc. in 1998. This annotation product was the first to appear commercially, and it embodied a design similar to the novel approach we had taken in developing JotBot, using a client process instead of a Java applet to monitor browser activity and retrieve and present annotations. Although NetNotions was built by a larger software development team, it underscores the shortened time it takes for an idea to go from research to commercial product using today's internet.

A general mechanism for annotating web documents can be the basis of a number of new and useful document management applications. We have developed an architecture to unobtrusively annotate web content and two concrete implementations of this architecture. Useful annotation capabilities could be introduced into today's Intranets and Internets without resorting to specialized infrastructure, by placing a simple intermediary function either in the browser (perhaps as an applet, as we have demonstrated) or at the server and controlling access to the intermediary. However, some relatively minor changes in current browser and server technology would make it much easier to extend their functionality. We have identified several desirable changes on each end, and would be gratified to see commercial products that adopt a more open stance toward extensions.

These conclusions are also summarized in our HICSS '99 conference paper "On Web Annotation: Promises and Pitfalls of Current Web Infrastructure"

Future Work

The technology we have implemented is now released.  Directions for further work include

• build vertical applications on top the core annotation function, e.g. by constructing a spider that automatically creates annotations to enhance or update information available on various topics.
• address scalability issues associated with large scale annotation systems:
• • finish extending the server software to use datagrams instead of IP connections
  • modify the data store to use a commercial DB instead of a file
  • modify the data store to use newsgroups for storing annotations to get wide-area replication
• explore using Trader architectures to dynamically locate and access appropriate annotation repositories. Communities of annotation users having similar interests but not be known to each other. An OMG trader-like service might be useful to match an annotations client to a relevant group's annotation repository.
• The OSA/Annotations architecture allows the stored form and the presented form of an annotation to be different. As appropriate, the system may lose information or transform the contents to a different format.
• evolve the capabilities of the systems we have built to exploit evolving web infrastructure.
• • define a generally flexible and extendible annotation structure using XML
  • use XML to build the annotation repository
  • using an XML browser, find ways to integrate annotations for non-XML pages with those for XML pages.