Network Working Group Mike O'Dell Internet-Draft UUNET Technologies October 1997 MIB Interoperability Testing 1. Status of this Memo This document is an Internet Draft. Internet Drafts are working documents of the Internet Engineering Task Force (IETF), its Areas, and its Working Groups. Note that other groups may also distribute working documents as Internet Drafts. Internet Drafts are draft documents valid for a maximum of six months. Internet Drafts may be updated, replaced, or obsoleted by other documents at any time. It is not appropriate to use Internet Drafts as reference material or to cite them other than as a "working draft" or "work in progress". To learn the current status of any Internet-Draft, please check the 1id-abstracts.txt listing contained in the Internet-Drafts Shadow Directories on ds.internic.net, nic.nordu.net, ftp.isi.edu, or munnari.oz.au. A revised version of this draft document will be submitted to the RFC editor as a Proposed Standard for the Internet Community. Discussion and suggestions for improvement are requested. This document will expire before March, 1997. Distribution of this draft is unlimited. 2. Abstract This document specifies the IESG's interpretation of the Internet Standards Process for the progression of SNMP MIB specifications to the Draft Standard level of maturity. It discusses the rationale for this interpretation and details the testing which is used to satisfy the advancement criteria. 2. Conventions used in this document In this document the words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" are used in strict accordance with governing Canon Law - see [RFC-2119]. O'Dell 1.6 [Page 1] Internet-Draft MIB Interoperability Testing November 1995 3. The Nature of the Problem The Internet Standards Process [BCP-XYZZY] requires that for a protocol to advance to the Draft Standard level, two genetically unrelated implementations must be shown to interoperate correctly with all features and options. There are two distinct reasons for this requirement. The first reason is to verify that the Technical Specification is adequately clear and accurate. This is demonstrated by showing that(at least) two different implementation efforts have worked from the specification and achieved interoperable implementations. The second reason is to discourage excessive options and "feature creep". This is accomplished by requiring interoperable implementation of all features, including options. If an option is not included in at least two different interoperable implementations, it must be remove before the specification can advance. In the case of a true protocol which specifies the "bits on the wire" exchanged by executing state machines, the notion of "interoperability" is reasonably intuitive - the implementations must successfully "talk to each other", exchanging "bits on the wire", while exercising all features and options. In the case of an SNMP Management Information Base (MIB) specification, exactly what constitutes "interoperation" is less obvious. It is precisely this notion of "MIB interoperability" that this document specifies. The difficulty is that there are a number of plausible interpretations of interoperability, all of which have merit but which have very different costs and difficulties in realization. The goal is to achieve an interpretation of interoperability and a testing model that strikes a balance between testing complexity and the competing desire to maximize the information generated by interoperability testing. 4. On The Nature of MIBs Compared to "state machine" protocols which focus on procedural specifications, a MIB is much more data oriented. To over- generalize, in a typical MIB the collection of data type and instance specifications outnumbers inter-object procedural or causal semantics by a significant amount. A central issue is that a MIB does not stand alone; it forms the O'Dell 1.6 [Page 2] Internet-Draft MIB Interoperability Testing November 1995 access interface to the instrumentation underneath it. Without the instrumentation, a MIB has form but no values. Coupled with the large number of objects even in a simple MIB, a MIB tends to have more of the look and feel of an API than a state machine protocol. 5. Some Alternatives The IESG debated a number of interoperability and testing models in formulating this specification. The following list is incomplete but captures the major plausible models which developed in the course of that discussion. 5.1 Basic Object Comparison Assume the requisite two genetically unrelated implementations of the MIB in an SNMP agent and an SNMP management station which can do a "MIB Dump" (extract the complete set of MIB object types and values from the agent implementation). Extract a MIB Dump from each implementation and compare the two to verify that both provide the complete set of mandatory and optional objects and that they are of the correct type. 5.2 Stimulus/Response Testing Proceed as in 5.1, but in addition, comprehensively exercise the two network elements containing the agent implementations to verify that all the MIB objects reflect plausible values in operational conditions. An even stricter interpretation would require that the MIB objects in the two network elements track identically given the identical stimulus. This does good testing on "read-only" or "monitoring" information obtained from the underlying instrumentation. 5.3 Full Coverage Testing This extends the notion of Stimulus/Response Testing to its logical extreme. Given that a MIB can be viewed as an API, the software engineering notion of full coverage testing could be applied to a MIB. This involves exercising all paths through the causal semantics and verifying that all objects change state correctly in all cases. 6. Discussion and Recommended Interpretation In a perfect world, Full Coverage or Stimulus-Response testing would materially increase the level of confidence in a MIB specification; this much we grant. However, experience in the software engineering real world makes it very clear that such confidence comes at an extremely high price, and that even with the most exhaustive testing, O'Dell 1.6 [Page 3] Internet-Draft MIB Interoperability Testing November 1995 it is often not clear what precisely has been demonstrated. We believe that both of those standards of evidence are beyond what can be reasonably accomplished in an operational sense, and is probably well beyond the ability to specify in any formal semantic sense. Therefore, the Operations and Management Area and the IESG adopt Basic Object Comparison as specified in section 5.1 above as the minimum demonstration of interoperability for advancing a MIB to Draft Standard status. This certainly does not preclude more aggressive testing, and the IESG reserves the right to take the advice of a Working Group who in strong consensus suggests that a particular MIB might require a stronger interoperability demonstration, but such advice should be quite rare. 8. Security Considerations Some will view this policy as possibly leading to a reduction in the level of confidence people can have in MIBs. Others will view this policy as a countermeasure to a significant denial of service threat. The O&M Area and the IESG view this as a competent engineering trade-off of multiple competing factors. 9. References [RFC-2119] "Keywords for use in RFCs to Indicate Requirement Levels", Bradner, March 1997 [BCP-XYZZY] "IETF Standards Process - The Standard Version", xxx, yyy, zzz 10. Author's Addresses Michael D. ODell UUNET Technologies, Inc. 3060 Williams Drive Fairfax, VA 22031 Email: mo@uu.net O'Dell 1.6 [Page 4]