Standards in a Time of Constant Change

by Karen Coyle

First published in the Journal of Academic Librarianship, v. 31, n. 3, pp 280-283

Standards have everything to do with change. They hold change in check by fixing certain parts of a technology. They also allow innovation to happen in a controlled way by creating an area of certainty around which change can happen. The process of change, however, is itself changing due to the increased rate of technology development, and this is having a profound effect on how we create and manage standards.

Standards, a Modern Need

The development of standards is a very modern process, and one that is increasing in importance as computer technology and networking take on primary roles in the functioning of our world. Although we talk about being an information society, in fact it might be more accurate to refer to ours as a communication society: the rate of exchange of data and data elements is astonishing. While data cannot replace some analog activities (there's no data equivalent to getting your dry cleaning done), for others bits and bytes are rapidly replacing the exchange of physical units, as in the cases of currency and of correspondence. The communication of bits and bytes requires consistency and precision, and thus requires standards.

Standards are so important that we have formalized the process of developing standards on both a national and international scale. This formal process, however, is being challenged by the rapid pace of technology development and by the increasingly ad hoc nature of computer technology development. We seem to have reached a key point in the history of standards, and one that will have a profound effect on the library community, which has a strong dependence on standards.

The Traditional View of Standards

In the traditional standards process, a defined community, such as manufacturers of flooring materials, would gather, decide that a standard is needed, create the standard, and then manufacture products to the standard for a number of years. The standard development process would be managed by an organization that the community as designed as its standards body. At regular intervals, such as every 5 years, the standards body would call members together to review the standard and make any modifications that were needed. Standards developed in this way would often be in place for decades, undergoing minor or even major modifications, but always defining the key processes of an industry and creating efficiencies for that industry as well as guarantees for its customers.

The library profession as been actively involved with this traditional standards process. As early as the 1870's, libraries were creating standards for their current technology: the card and the content that it held. The MARC record standard is more modern but it followed a traditional standards process. Developed under the aegis of the American National Standards Institute (ANSI), the MARC record format standard was originally defined in the late 1960's before any computer systems were processing library catalog data. Defining and maintaining the semantics of the MARC record has been the responsibility of the Library of Congress since that time. Changes are made to the content of the record using a painstaking process that requires the collaboration of representatives of libraries and archives. Although changes continue to be made, the MARC record process is an example of the previous generation of standards.

Standards Today

The main constant of our computer technology environment is change, and that means that standards and the process that creates and supports them must be part of that change. Today's standards process can be considerably different from the traditional one, especially in the information industries. Rather than beginning with a standard, new technologies often begin when a developer sees a local need and does some preliminary development, usually of a software product. The developer shows this product to others, some of whom also see the need, and who then gather in an ad hoc community of interest. The development may not go beyond this stage, but if the product turns out to be of interest to many, it may reach the point that it is requested as a feature in vendor products. Once it is of interest to vendors, standardization becomes essential so that all vendor systems can exchange data. The new technology goes through a standardization process, although this is often in parallel with deployment of one or more early versions of the software or service. Once approved, the standard enters process of near-constant change as new communities discover uses for it and as systems evolve.

Traditional Standards ProcessModern Standards Process
Needs assessmentFelt need
Standard developmentPreliminary development
Product developmentAd hoc community formation
Maintenance phaseWider need expressed, community formed
Standard development & product development in parallel
Maintenance phase
Fig. 1 Standards Process

Note that in the modern standards process, there is often a need for community formation around a standard, whereas in the traditional process the community exists before the standard is begun. In that latter, the community was pre-defined and might remain stable over a significant period of time. Today communities form often in an ad hoc fashion around a particular technology. The members of that community may have little in common except that they use the same technology. A good example of this are the communities that form around the XML standard. The "ML" in XML is Markup Language, and XML is itself a tool to create markup languages in any realm. Groups have gathered to create a number of different markup languages, such as MathML for mathematical notation, or Data Center Markup Language (DCML), an XML-based specification for representing the contents of data centers. MathML was developed by a group that consisted of publishers, software makers that specialize in mathematical software, and universities. DCML's community is made up mainly of companies that specialize in complex enterprise server configuration. Once the language is stable, these communities may disband if they have no need to develop the standard further. In other cases, communities may continue as maintenance organizations for the standard as long as it is being actively used.

The idea that one can wait five years to review a standard is long past, as is the concept that a standard might serve for decades with only a few tweaks. As a matter of fact, by the time the formal standards process is completed, which often takes two to three years, the standard in question can be out of date or at least on the wane. Technology moves too rapidly for the formal standards process to be relevant. What we need is a way to create standards that can change. But how can it be a standard if it keeps changing? To understand this, we need a way to characterize different levels of standards.

The Standards Pyramid

There are standards that stay steady for years on end, and there are others that move forward rapidly with upgrades, modifications, and new implementations. The difference is often a matter of where they are on what I think of as the "standards pyramid," with a wide base of general standards and a "tip" of specific applications. This concept also has a traditional view, and a more modern view. The bottom layer is made up of very basic standards that everything else depends on. This is where we define bits and bytes, character sets, programming languages, and communications protocols. All computer technology uses some of these basic standards, although these are so fundamental that few of us have to think about them when we develop new technology.

The next layer is what I think of as reusable applications or tools. These are applications like the HTTP of the World Wide Web, or XML. XML is not usable in itself, but it can be used to create applications and to create standards for applications. XML is what makes the widely used language of HTML possible. XML dictates that every field must have a start tag (between "<" and ">") and an ending tag (between "</" and ">") but the tags themselves can be anything you wish. HTML defines sets of tags for documents, like "<p></p>" for paragraphs and "<ol></ol>" for ordered lists. There are hundreds of standard languages using the rules that XML provides, and many more local applications that take advantage of its flexibility.

Standards pyramid with HTML

Two examples of standards in this layer from our environment are Z39.50 and the OpenURL standard. Both of these are prime examples of the use of a standard to hold some things constant so that others can change. Both can be, and have been, incorporated into a variety of applications, and are supported by different profiles that are used in applications. The fundamental structure of the MARC21 record, as defined in the standard ANSI/NISO Z39.2, is another potentially reusable application, although it hasn't been reused much beyond its primary use in the library community.

Standards pyramid with MARC21

The final and top layer in the traditional view I call standard applications. This is where the standard actually includes some semantics or content. MARC21, with all of its tags and subfields and the rules for input, is a standard application. The OpenURL profile that carries article citation metadata to a resolver service is a standard application. There are also identifiers that are standard, such as the ISBN and ISSN, and the DOI. All of these are at the level where there is an actual use of the data that is defined in the standard, and the data has meaning for users of the technology.

The most rapid change happens in the applications layer. This is the layer that gets things done for users and where vendors might compete by offering newer or improved products. The least change takes place in the basic standards layer. Those standards are incorporated into all of our software and hardware and any change at that layer wreaks havoc in the installed technology base. The middle layer is the buffer between the slow moving basic technology and the rapid pace of applications. If well-designed, that middle layer can provide some stability for the application layer along with the flexibility to allow some change to take place.

The MARC format, an example of a standard that is very familiar to us in the library profession, is based on the very stable technology of the ASCII character set. Although MARC is now moving into the use of the Universal Character Set (also known as "Unicode"), ASCII is such a basic standard that the UCS has been developed in a way that will not interfere with data in ASCII, making much of the MARC standard upwardly compatible with the character set change. MARC itself is a record structure defined by an ANSI/NISO standard, Z39.2, and also registered as an international standard as ISO 2709. This basic record structure has not changed since its creation in the mid-1960's. What has changed is the application layer, the standard known as MARC216. Programs that can read the Z39.2 structure can read records created in 1970 as well as records that we create today. Making use of the content of the record requires keeping up with the latest changes in MARC21. So there is both stability and change.

The New Pyramid and Profiles

With the need for greater flexibility and more rapid development of new applications, this model is being replaced by one that models applications through the creation of profiles. You can think of profiles as being made up of interchangeable parts that are defined in the standard, a kind of "one from column a, two from column b" approach. Any particular combination of the parts becomes a profile. In our pyramid diagram, the middle layers define the elements that can make up a profile. Either the standard itself or a maintenance function added onto the standard defines how profiles will be made available to potential users. Some standards communities have developed ways that developers can register machine-readable profiles so that others can easily write applications that can create and accept compatible data.

The Z30.50 protocol is an example of a use of profiles. Z39.50 provides a way to send a database search from one system to another using a standard method of interaction. It also provides a standard way to define the search that is to be done. Because not every database will support every kind of search, and because new types of searches can develop over time, the communities using Z39.50 define particular uses of the standard through profiles that particularly meet their needs. Profiles can be used to define types of searches, formatting of results, etc. As an example, the US National Z39.50 profile, standardized as NISO Z39.89, has elements for searching, browsing and displaying records from library catalogs and is often used for searching across multiple catalogs. The European ONE-2 profile also allows for catalog searching but it has particular elements for interlibrary loan functions that are suitable to the ILL systems in the European document sharing environment.

OpenURL is another example of a standard that relies on profiles. It provides a general structure that can encapsulate nearly any kind of data elements that describe a document or file or even a network service, but it doesn't define the data elements within that structure. Instead, the content of the OpenURL is defined in a profile. The structure of the profile is also part of the standard, so programs can be written that understand OpenURL but that must be adapted to the details of individual profiles. The basic OpenURL structure occupies our layer of reusable tools, and allows the creation of application standards in our top layer.

There are advantages to the method of defining standards in terms of profiles. One of these is that applications do not need to implement every element of the standard, only those that relate to the profile they are interested in. So, using MARC21 as an example, every library systems vendor must implement every field and every data element in the MARC21 record. With profiles, some vendors that serve defined communities could implement only those elements relevant to that community. The implication of this is that a single standard can satisfy the needs of multiple communities through the flexible mechanism of profiling. This means that one community might be able to "borrow" a standard developed by another without having to go through the standards process itself. The disadvantage is that the users of a particular standard can be a rather hodge-podge bunch and might not have enough of a shared picture of the world to work together on maintenance of the standard. This is where a strong standards organization can make a world of difference.

What This Means For Standards Organizations

There are a number of ways in which the change to the nature of standards and the standards process affects the kinds of standards organizations that arise to meet that need. The traditional standards organization was based on a single community of interest. If the community was one with a strong identity, such as our own library community, the organization had a clear idea of the community's needs and requirements. Now standards often serve more than one identifiable community, or even something as amorphous as "the community of people who might want to use this standard." This may work fine in the short term, but it may mean that there is no organization that has a real vision of the future direction of development for a particular community. It also means that the standards process itself can be more difficult to manage because of the lack of shared values and a shared vision of the direction the standard should take.

Because it is often the case that no one standards organization represents the community of interest, organizations are joining in consortia for the purpose of developing standards. Groups like the Organization for the Advancement of Structured Information Standards (OASIS) and the World Wide Web Consortium (W3C) are examples of this. These organizations each represent a particular standards suite that is used by a range of companies and institutions that in other arenas do not constitute a community of interest. A key difference between these consortia and the traditional standards organization is that the latter was often defined as national in scope. Information technology standards are valid beyond national boundaries, especially those technologies based on the Internet. Standards development process today has to be international in scope and the consortia reflect this.

Finally, the range of development activities that a standards organization must be involved with has changed. Traditional standards organization created standards, then managed some periodic maintenance. Today's organization has to be involved in the preliminary process to identify ad hoc developments that need to be brought into the standards process. They have to manage a constant and ongoing update process, through registries. They also have to participate in the activities of a variety of organizations that can influence the standard and its future. This places a burden on the standards organization to be active in many arenas at once, not just its own processes.

Next: The State of Library Standards

In the next issue, we'll look at library standards and how they fit in this new standards arena. What are the current most important library standards? Are we well served by our standards organizations? Does it even make sense today to talk about a "library" standard? Should we be part of the larger standards development community, such as W3C or IETF? These are the questions that the library standards community is asking itself today.

The copyright in this article is NOT held by the author. For copyright-related permissions, contact Elsevier Inc.