Considerations for NENA Working Groups in Designing Web Services for Maximum Consistency
NENA DS-Interface Specification Development WG[1]
October 20, 2020
Abstract
This document provides neutral information regarding certain aspects of web services, which are standardized interfaces that use web (HTTP) protocols as transport. In addition to describing these aspects, it also provides guidance for maximizing consistency among NENA web services.
Introduction
Standards commonly include functionality performed by cooperating entities using a defined interface. For example, NENA i3 specifies a logging function where various elements generate log events that are created and stored by a logging server, and entities are able to retrieve log events from the logging service. In general, such functionality is specified using a network protocol (which allows the entities to be located on the same machine, on different machines, even on different machines in different networks, and to be provided by different vendors). Designers of such protocols may choose to create a new protocol for their purpose (which might be an entirely new protocol or a modification of an existing protocol) or to re-use an existing protocol. When a protocol is reused, the standard specifies how the information for the functionality is conveyed using the existing protocol. Various models are possible, although often such services use a concept of a request and a response, and often the side making a request is referred to as the client, and the side receiving the request and sending a response is referred to as the server. Protocols may be directional (one side initiates requests and the other side responds) or not (either side may send data to the other at any time), and may be synchronous or asynchronous. In synchronous protocols, messages are only sent at defined instances, for example, in synchronous client/server protocols, servers only send messages to clients in response to a request; in some protocols, each response can only result in one response. Asynchronous protocols allow messages to be sent at any time or for multiple responses to be sent to a single request. For example, an HTTP[2] server sends one response to each request; a SIP[3] server may send intermediate responses before sending a final response to an initial request, and once a SIP session has been established, either side may send messages to the other at any time; XMPP[4] was explicitly designed to allow endpoints to send messages to other endpoints at any time.
In recent years, a common choice for service designers has been to use HTTP over TLS[5] (referred to as HTTPS), with data needed for the functionality conveyed as XML or JSON[6] objects in the request and response messages. One reason this is often the choice is the wide availability of and familiarity with tools that handle HTTP over TLS with XML[7] or JSON objects. Such tools allow for automating aspects of the implementation. When such functionality uses HTTP, it is referred to as a web service (that is, a service implemented using web protocols and techniques), and the web service can be described as being layered on top of HTTP.
NENA work groups (WGs) that design web services typically need to decide how various aspects of the interfaces function. Certain important aspects are described below.
Web Service Design Aspects
Web Services HTTP Status Codes
Description:
Web service functions need to convey the status of a request, most fundamentally, if the request succeeded or failed (although requests might succeed with warnings, and other nuances are possible). HTTP itself has a three-digit status code that is conveyed in responses. A web service that uses HTTP to convey information can convey the result of a request in different ways, as described below:
1. Strict Layering:
At the HTTP layer, status codes only indicate the status of the HTTP layer. The status of the actual web service operation is indicated in a web services object contained in the response body, or in a header field of the response.
E.g., in most cases, an HTTP 200 OK result is returned in the HTTP response, while the returned web services data object, or in a header field of the response, carries the web service status code (typically as an integer or as a constrained string, e.g., limited to digits and dots) and a web services text response (as a string).
Pros: From an architectural or philosophical view, this is the cleanest approach. It allows an interface to maintain strict layer separation. Only standard HTTP status codes are used. It provides consistency, in that the result of a request is always returned the same way, reducing the potential for errors.
Cons: A web services object or response header field must always be included in the response body, even if, aside from the response code, there are neither parameters nor data to be returned. A client must always parse the returned object or header field to determine the result of a requested operation. It adds complexity in that the returned object or header field in most cases must be consulted to determine if an operation succeeded or what steps a client needs to take to handle an error.
2. Custom:
The HTTP response code indicates the status of the actual web service operation, including the status of the HTTP transaction itself where relevant. IANA-registered HTTP status codes are used where these codes are an exact or close match to the status of the web service request. Non-overlapping custom status codes are used where no IANA-registered HTTP status code is a good fit. The custom codes are collected and standardized for use within NENA.
Pros: Simplicity: the status of the request is always conveyed using the HTTP status code. Implementations can make decisions on next steps based on the status code without needing to parse the result message.
Cons: Results are limited to three digits using the HTTP structure where the first digit indicates success, temporary failure, permanent failure, etc., which can make it difficult to convey sufficient detail or nuance. Philosophically, it mixes the results of the web service with the HTTP used for transport.
3. Mixed:
If an IANA-registered HTTP status code is appropriate and an exact or close enough match for the status of the web service request, that status code is returned as the HTTP response to indicate the status of the actual web service operation. When there is no IANA-registered HTTP status code that is a good fit, a generic HTTP response code is returned, and the response also carries a web services object or header field that contains the actual status code (typically as an integer or as a constrained string, e.g., limited to digits and dots) and text response (as a string). A web services object or response header field is only returned when necessary.
Pros: Web services requests are expected to succeed most of the time, and when they fail, it is most likely because of an authentication or other common situation that fits a standard HTTP response code. Hence, this approach means that in many cases, the client need only check the returned HTTP response
Cons: The status of a request is sometimes indicated by the HTTP status code and sometimes by a field in an object or a header field. This may increase the risk of an implementation failing to check the returned object or header field.
Guidance:
For maximum consistency among NENA web services, use custom error codes (method 2 above), with the codes registered in the NENA Status Code Registry
Request and Response Parameters
Description:
Web service requests typically contain parameters that specify details of the request, and responses often contain parameters to convey requested data, details of a failure, or other information. There are three ways to pass parameters:
1. Request parameters may be passed in the URL. There are two mechanisms for this:
a. The parameters (often identifiers), as values (or sometimes as names and values), may be components of the path. E.g., the following URL operates on the “widgets” resource with the identifier “3”; the identifier parameter is a component of the path:
https://foo.example.org/interface/widgets/3
b. The parameter may appear at the end of the path, delineated by a question mark or semicolon, e.g., in the following URL, the parameter name (“widgetID”) and value (“3”) are passed using “?” syntax:
https://foo.example.org/interface/widgets?widgetID=3
2. Parameters may be encoded as HTTP header fields of requests or responses, e.g., the following example HTTP request fragment contains (at the end) a header field “Web-Service-widgetID” with a value of “3”:
GET / HTTP/1.1
Host: foo.example.org
User-Agent: WizzBang CPE Magic 43
Accept: application/json
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Web-Service-widgetID: 3
3. Parameters may be encoded within a body part of a request or response.
HTTP messages consist of a header, containing fields, and optionally, a body consisting of a MIME[8] type. A MIME type may be multipart, containing one or more child MIME parts. The term “body part” is typically used to refer to any of these MIME type elements, even when the body is a single MIME type.
When encoding parameters within a body part (method 3), virtually any MIME type could be used to contain the parameters. For example, a web service could use a “text/plain” body part, defining a text-based syntax for encoding parameters, or could use a “multipart/form-data” or “application/x-www-form-urlencoded” body part as if sending HTML <form> data. A web service could create and register its own MIME type, with parameters structured using JSON, XML, or any other format. Since web services typically make use of JSON or XML object encoding, parameters are most commonly passed as one or more JSON or XML objects, i.e., as one or more body parts of “application/json”, “text/xml”, “application/xml”, or one of the many more specific MIME types with a subtype containing “+json” or “+xml” (indicating JSON or XML encoding).
Note that passing a body in a GET “has no defined semantics,[9]” although some tools and systems support it. Since NENA web services operate within the constrained context of NENA-defined systems, doing so is less of a concern than it would be with the web in general, where endpoints and intermediate systems (such as proxies) might not support it.
The URL (method 1) and header field (method 2) mechanisms are primarily suitable for scalar values (such as strings, numbers, and Booleans), as opposed to complex objects (such as arrays). While complex objects could be encoded into header fields, in practice, the most reasonable solution is to encode them as an object in the body (such as a JSON or XML object).
When passing parameters in a URL (method 1), there are length and encoding considerations. All data in a URL must be encoded to be URL-safe (e.g., a string parameter value that contains a question mark or semicolon must be encoded). The total length of a URL is restricted, and in some environments, there may be constraints or considerations on the size of message headers or the entire message.
For the HTTP header field mechanism (method 2), the use of “X-“ header fields is common, but discouraged by the IETF.
Identifiers can be considered a special case. The REST[10] style encourages a conceptual model in which operations are performed on resources; when there is a simple identifier for a resource, making it an element of the path is a common approach that facilitates using the HTTP verbs in what are generally considered appropriate ways (i.e., GET to retrieve, POST to create, PUT to update, and DELETE to delete). When complex or sizeable parameters need to be passed in addition to an identifier, there will be a data object in the HTTP request body anyway, so passing the identifier in the path means that most parameters are in the body while some are in the path, which can needlessly complicate the mechanism. When only simple parameters are needed, the use of path elements plus URL question mark and semicolon parameters is an option (subject to URL length and encoding constraints).
Guidance:
For maximum consistency among NENA web services, if the interface operates on resources that have a client-assigned identifier, the identifier should be provided using path elements (method 1, sub-method a above). Where parameters are a small number of simple quantities, the parameters should be provided as URL parameters (method 1, sub-method b above). Otherwise, parameters should be provided in an object in the body of the request (method 3 above). For maximum consistency, if a request body is needed, the request SHOULD NOT use the HTTP GET verb (POST is commonly used in such cases).
HTTP Verbs
Description:
In a web service request, a client sends an HTTP request to a server, and receives a response. HTTP defines a number of request methods, often called verbs. The following HTTP verbs are commonly used when designing web services:
1. GET
2. PUT
3. PATCH[11]
4. DELETE
5. POST
Section 2 (Request Parameters) describes how parameters are passed.
The amount of time that elapses from when a server receives a request until it transmits the response will of course vary depending on multiple factors, including server load or stress, resource storage (e.g., local versus remote database, database type, etc.). In general, operations that access or retrieve data tend to be overall quicker than operations that write data, although server-specific conditions are typically more significant. Network conditions and intermediate systems obviously can affect round-trip times. Some interface designs accommodate the possibility that some operations might take an extended amount of time by allowing for polling or call-backs. When invoking an interface that does not support polling or call-backs, client implementations must wait for a response or until a maximum wait time (timeout) is reached. Typically, clients either:
· Transmit a request and wait for a reply or a timeout, with a timeout treated as an error, or
· Transmit an operation and perform other tasks while waiting for a reply, with a timeout treated as an error.
The former is sometimes termed “blocking” or “hanging” on a response. The latter may be implemented as a main processing loop that invokes code for various events that may occur, one of which is receiving a response to a previously sent request.
An interface may be designed for extended response times by use of a poll or callback facility.
A poll facility typically has the server send an immediate response that contains a token. The client subsequently sends a request to the same or different URL and passes the token; the response contains either the result of the initial request or an indication that the result is not yet available, in which case the client either tries again or determines that too much time has elapsed and treats the situation as an error. (The initial request may be defined such that either a token or initial parameters are passed, or there may be two URLs defined, one for the initial request and one for subsequent status polls.)
A callback facility is typically defined by having a parameter in the initial request that contains a callback URL. The response to the initial request indicates if there are any immediately detected errors; if not, when a final response is available, the server (acting as a client) sends an HTTP request to the URL specified in the callback parameter.
Poll and callback are most commonly used with POST, GET, PUT, and PATCH, less so with DELETE, and rarely with other verbs.
In cases where certain services are expected to have non-immediate responses, there are several typical approaches to such services:
1. Invoking the service always uses polling;
2. Invoking the service always uses callback;
3. Invoking the service optionally uses polling;
4. Invoking the service optionally uses callback;
5. Two versions of the service are defined, one that uses polling and one that does not;
6. Two versions of the service are defined, one that uses callback and one that does not;
A service that always uses polling defines a token as a mandatory parameter of the response; after sending the request and receiving the response with the token, the client must repeatedly send a request at intervals, passing the token, to receive the final response if available or a status that it is not yet available. Responses other than the final one may include a parameter with a suggested interval value, or a minimum interval value, for the client to use when making the next request.
In a service that optionally uses polling, a token is defined as an optional parameter of the response; the server includes it if it expects a delayed response, otherwise it sends the final response.
A service that always uses callback defines a callback URL string as a mandatory parameter of the request; when the final response is available, the sends a request to this URL with the final response.
In a service that optionally uses callback, a callback URL string is defined as a mandatory parameter of the request; the server indicates in the initial response if it contains the final response or not; if not, the server sends a request to the callback URL when the final response is available.
When two versions of a service are available, one that uses callback or polling and one that doesn’t, the version that doesn’t specifies that the response to the request is the final response, regardless of how much time it might take.
In some situations, the response to a request may take a variable but typically small amount of time, such as no more than tens or hundreds of seconds, while in other cases, a response may take hours, days, or even weeks or even months. In the latter case, callback is usually used, to avoid situations where a client may need to repeatedly poll for significant amounts of time, wasting resources.
Note that the GET verb can be viewed as clumsy when a resource is being updated, deleted, or created; the PATCH verb is inarguably clumsy when a resource is not being updated; and the DELETE verb is inarguably incorrect when a resource is not being deleted.
Guidance:
Note that the NENA specification that defines the largest number of web services is i3. All i3 web services return immediate final responses except for the Discrepancy Report service, where a final response depends on human action, which could take minutes to months, and therefore uses non-optional callback (method 2 above).
For maximum consistency among NENA web services, use the most appropriate HTTP verb. When web services have non-immediate final responses, use non-optional call back (method 2 above).
Resources
Description:
Web service interfaces often store and act on resources (e.g., persistent objects such as documents). Conceptualizing the target of web service requests as resources and the HTTP request methods as verbs can make the use of specific HTTP requests more natural.
Resources typically have an identifier, allowing them to be referenced. A client might store a new resource and subsequently request (read), update (replace or in some models partly replace), or delete the resource. When a web service request creates a new resource, the resource’s identifier can be assigned by the server (e.g., the entity that stores it) or by the client (the entity that causes its creation). In the example of a client making a request of a logging service to store a new log event, the identifier for the log event might be assigned by the client and passed in the request with the log event data, or might be assigned by the logging server and returned in the response to the request. Resource identifier assignment can be described as following one the following models:
1. Client: Resource identifiers are always assigned by the client side;
2. Server: Resource identifiers are always assigned by the storage entity (server side);
3. Varied: ID assignment is based on the specific web interface.
When a server assigns a resource identifier, the new identifier value is typically returned in the response to the request that created the resource. If the response contains other values (e.g., there is an object in the body), the identifier value may be included; otherwise, the Location header field of the response may contain a URL for the new resource.
Section 2 discusses passing the resource identifier in the path.
Guidance:
For maximum consistency among NENA web services, use varied ID assignment. Generally, when identifiers are assigned automatically or there are multiple items, server-assigned identifiers are preferred. When identifiers are assigned by a human, or are passed to the server during resource creation embedded in a structure, client-assigned identifiers are preferred.
For consistency, it is RECOMMENDED that resource names in resource URIs be nouns, and that JSON schema be compliant with NIEM JSON guideline. For an example of a resource name as a noun, see the use of “widgets” in Section 2 method 1.
Server State Assumptions
Description:
1. The server is assumed to be stateful (it maintains state for the same client for a series of transactions);
2. Web service transactions are assumed to be stateless at the web services level (TLS may be persistent but each web service transactions is performed independently of any prior or subsequent transactions).
Guidance:
For maximum consistency among NENA web services, use stateless transactions.
Versioning
Description:
APIs are revised over time. Often a version change is backwards compatible, but not always. For example, adding elements to a request or response is backwards compatible if implementations ignore elements they don’t understand. APIs usually have a versioning mechanism to deal with these issues.
Both the client and the server may support one or more versions. To achieve compatibility there needs to be some mechanism to discover what the other end supports, and to respond appropriately. Since it is common for multiple versions to be supported at either end, a way to specify which version is being used for a given transaction is also needed.
Five mechanisms are commonly used for this purpose:
1. A header field specifies what version is supported. Often this is the Accept header field but sometimes a custom header field is used. This is the same as passing the version as a parameter using the HTTP header as described in Option 2 of Section 2 (Request and Response Parameters).
2. The version is specified in the path. The following example requests the resource “Policies” of version “v1.3” of the “PolicyStore” service of an API provided by the host “services.example.com”:
https://service.example.com/PolicyStore/v1.3/Policies
3. A version entry point is included in the API for the service or server. A client performs a GET on this entry point and receives a list of versions supported at the server. For example, a GET on the URL https://service.example.com/PolicyStore/Versions might return “1.0, 1.5, 2.0, 3.0”.
The “versions” entry point can be defined per server or per service. Defining “versions” as per server implies a simpler set of web services where it is expected that an individual server implementation will monolithically support all entry points for all services. Defining a “versions” entry point for each service allows more complex situations where a server build supports different versions per service. As an example, a vendor might have a major release that supports version 6 of all services, then in a subsequent point release update one service to version 7, etc., as a means of getting support for later versions into deployment faster. Defining “versions” as per service means that even though the same server can provide multiple services, clients must send a request to the “versions” entry point of each service before using the service; version information is allowed to differ between services. Either way, it’s important to note that version information might change between requests, as a server could fail-over or be upgraded in between requests.
4. A service discovery mechanism has version information in it.
5. New endpoints are defined with new names when non-backwards compatible changes are introduced. For example, the service api.example.com/PolicyStore/Policies might be deprecated in favor of the new service api.example.com/PolicyStore/ManagePolicies.
Note that the header field mechanism (item 1) typically specifies what is supported at the client, while the others usually specify what is supported at the server. When the query doesn’t have non-backwards-compatible extensions, the header approach allows any client to construct a query that is acceptable for any server where there is at least one common version, and the server can determine from the header what version response the client can accept.
The path mechanism (item 2) allows the client to specify the version it is using, but unless one of the other server version mechanisms are available to inform the client which versions are supported at the server, the client has to guess, potentially requesting an unsupported version, receiving an error that the URL doesn’t exist, trying another URL with a different version, and so on.
On the other hand, the versions resource (mechanism 3) and service discovery versioning mechanism (mechanism 4) don’t have a way to for the client to tell the server which version it wants to use. Hence, the path mechanism (item 2) in combination with the versions service (mechanism 3) or service discovery (mechanism 4) is a common choice.
Guidance:
For maximum consistency among NENA web services, pass the version as a path element (item 2 above). In addition, specify a “Versions” entry point for each web service, which returns the versions supported by the server. For maximum consistency, specify a “Versions” entry point that is identical to that in STA-010.
Tooling
Description:
As mentioned in the Introduction and elsewhere, software development commonly uses tools to automate some of the code development, verification, or other aspects. A number of tools have been designed for HTTP protocols, which (as mentioned in the Introduction) is one reason HTTP is often chosen as the layer on which to build services. When designing a service, especially a web service, one consideration may be the degree of compatibility of the design with common tools. For example, service designers might choose to define their services using an OpenAPI[12] format, to be suitable as input into various tools.
Guidance:
For maximum consistency among NENA web services, use the Swagger Hub tool, which supports Open API. NENA has a limited license for this tool; contact the NENA Technical Issues Director for access. Consult the i3 Policy Store and Logging web services for examples of how to structure the YAML file (which defines the web service) to maximize consistency. Also note that the swagger.io[13] tool is available for free and can be used to develop and validate the correctness of an API.
[1] https://dev.nena.org/communities/community-home?CommunityKey=041f8444-7e41-423f-94b7-4ad0f00acb5a
[2] Hypertext Transfer Protocol (HTTP), RFC 7230, https://tools.ietf.org/html/rfc7230.
[3] Session Initiation Protocol (SIP), RFC 3261, https://tools.ietf.org/html/rfc3261.
[4] Extensible Messaging and Presence Protocol (XMPP), RFC 6120, https://tools.ietf.org/html/rfc6120.
[5] Transport Layer Security (TLS), RFC 5246, https://tools.ietf.org/html/rfc5246.
[6] The JavaScript Object Notation (JSON) Data Interchange Format, RFC 8259, https://tools.ietf.org/html/rfc8259.
[7] Extensible Markup Language (XML), World Wide Web Consortium, Recommendation REC-xml-20081126, November 2008, http://www.w3.org/TR/2008/REC-xml-20081126.
[8] Multipurpose Internet Mail Extensions (MIME), RFC 2046, https://tools.ietf.org/html/rfc2046.
[9] Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content, RFC 7231, https://tools.ietf.org/html/rfc7231.
[10] Representational state transfer (REST), a style in which HTTP usage is constrained in certain ways, to optimize and simplify the use of HTTP. There is no single, authoritative definition of REST, but it is generally agreed that it entails a model in which HTTP verbs are used to act on resources.
[11] “PATCH Method for HTTP”, RFC 5789, https://tools.ietf.org/html/rfc5789
[12] https://www.openapis.org, https://swagger.io, etc.
[13] https://editor.swagger.io/