INTERNET-DRAFT Charles H. Lindsey Usenet Format Working Group University of Manchester July 2001 News Article Format Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC 2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Abstract This Draft defines the format of Netnews articles and specifies the requirements to be met by software which originates, distributes, stores and displays them. It is intended as a standards track document, superseding RFC 1036, which itself dates from 1987. Since the 1980s, Usenet has grown explosively, and many Internet and non-Internet sites now participate. In addition, this technology is now in widespread use for other purposes. Backward compatibility has been a major goal of this endeavour, but where this standard and earlier documents or practices conflict, this standard should be followed. In most such cases, current practice is already compatible with these changes. [The use of the words "this standard" within this document when referring to itself does not imply that this draft yet has pretensions to be a standard, but rather indicates what will become the case if and when it is accepted as an RFC with the status of a proposed or draft standard.] C. H. Lindsey [Page 1] News Article Format July 2001 [Remarks enclosed in square brackets and aligned with the left margin, such as this one, are not part of this draft, but are editorial notes to explain matters amongst ourselves, or to point out alternatives, or to indicate work yet to be done.] [Please note that this Draft describes "Work in Progress". Much remains to be done, though the material included so far is unlikely to change in any major way.] Table of Contents 1. Introduction .................................................. 5 1.1. Basic Concepts ............................................ 5 1.2. Objectives ................................................ 6 1.3. Historical Outline ........................................ 6 1.4. Transport ................................................. 6 2. Definitions, Notations and Conventions ........................ 6 2.1. Definitions ............................................... 7 2.2. Textual Notations ......................................... 8 2.3. Relation To Mail and MIME ................................. 10 2.4. Syntax Notation ........................................... 10 2.5. Language .................................................. 13 3. Changes to the existing protocols ............................. 13 3.1. Principal Changes ......................................... 13 3.2. Transitional Arrangements ................................. 14 4. Basic Format .................................................. 15 4.1. Syntax of News Articles ................................... 15 4.2. Headers ................................................... 16 4.2.1. Names and Contents .................................... 16 4.2.2. Header Properties ..................................... 18 4.2.2.1. Experimental Headers .............................. 18 4.2.2.2. Inheritable Headers ............................... 18 4.2.2.3. Variant Headers ................................... 18 4.2.3. White Space and Continuations ......................... 19 4.2.4. Comments .............................................. 20 4.2.5. Undesirable Headers ................................... 20 4.3. Body ...................................................... 21 4.3.1. Body Format Issues .................................... 21 4.3.2. Body Conventions ...................................... 21 4.4. Characters and Character Sets ............................. 23 4.4.1. Character Sets within Article Headers ................. 23 4.4.2. Character Sets within Article Bodies .................. 24 4.5. Size Limits ............................................... 25 4.6. Example ................................................... 26 5. Mandatory Headers ............................................. 27 5.1. Date ...................................................... 27 5.1.1. Examples .............................................. 28 5.2. From ...................................................... 28 5.2.1. Examples: ............................................ 28 5.3. Message-ID ................................................ 29 5.4. Subject ................................................... 29 5.4.1. Examples .............................................. 30 5.5. Newsgroups ................................................ 31 C. H. Lindsey [Page 2] News Article Format July 2001 5.5.1. Forbidden newsgroup names ............................. 35 5.6. Path ...................................................... 36 5.6.1. Format ................................................ 36 5.6.2. Adding a path-identity to the Path header ............. 37 5.6.3. The tail-entry ........................................ 38 5.6.4. Delimiter Summary ..................................... 38 5.6.5. Suggested Verification Methods ........................ 39 5.6.6. Example ............................................... 40 6. Optional Headers .............................................. 41 6.1. Reply-To .................................................. 41 6.1.1. Examples .............................................. 41 6.2. Sender .................................................... 42 6.3. Organization .............................................. 42 6.4. Keywords .................................................. 42 6.5. Summary ................................................... 42 6.6. Distribution .............................................. 43 6.7. Followup-To ............................................... 44 6.8. Mail-Copies-To ............................................ 44 6.9. Posted-And-Mailed ......................................... 45 6.10. References ............................................... 46 6.10.1. Examples ............................................. 46 6.11. Expires .................................................. 47 6.12. Archive .................................................. 47 6.13. Control .................................................. 48 6.14. Approved ................................................. 48 6.15. Replaces / Supersedes .................................... 49 6.15.1. Syntax and Semantics ................................. 49 6.15.2. Message-ID version procedure ......................... 50 6.15.2.1. Message version numbers .......................... 51 6.15.2.2. Implementation and Use Note ...................... 52 6.15.2.3. The Message-Version NNTP extension ............... 54 6.15.2.4. Examples ......................................... 54 6.16. Xref ..................................................... 55 6.17. Lines .................................................... 57 6.18. User-Agent ............................................... 57 6.18.1. Examples ............................................. 58 6.19. Injector-Info ............................................ 58 6.19.1. Usage of Injector-Info-header-parameters ............. 60 6.19.1.1. The posting-host-parameter ....................... 60 6.19.1.2. The posting-account-parameter .................... 61 6.19.1.3. The posting-sender-parameter ..................... 61 6.19.1.4. The posting-logging-parameter .................... 61 6.19.1.5. The posting-date-parameter ....................... 61 6.19.2. Example .............................................. 61 6.20. Complaints-To ............................................ 61 6.21. MIME headers ............................................. 62 6.21.1. Syntax ............................................... 62 6.21.2. Content-Type ......................................... 62 6.21.2.1. Message/partial .................................. 63 6.21.2.2. Message/rfc822 ................................... 63 6.21.2.3. Message/external-body ............................ 64 6.21.2.4. Multipart types .................................. 64 6.21.3. Content-Transfer-Encoding ............................ 65 6.21.4. Character Sets ....................................... 66 C. H. Lindsey [Page 3] News Article Format July 2001 6.21.5. Content Disposition .................................. 67 6.21.6. Definition of some new Content-Types ................. 67 6.21.6.1. Application/news-transmission .................... 67 6.21.6.2. Message/news withdrawn ........................... 68 6.22. Obsolete Headers ......................................... 68 7. Control Messages .............................................. 69 7.1. The 'newgroup' Control Message ............................ 69 7.1.1. The Body of the 'newgroup' Control Message ............ 70 7.1.2. Application/news-groupinfo ............................ 71 7.1.3. Initial Articles ...................................... 72 7.1.4. Example ............................................... 73 7.2. The 'rmgroup' Control Message ............................. 74 7.2.1. Example ............................................... 74 7.3. The 'mvgroup' Control Message ............................. 74 7.3.1. Single group .......................................... 74 7.3.2. Multiple Groups ....................................... 75 7.3.3. Examples .............................................. 76 7.4. The 'checkgroups' Control Message ......................... 77 7.4.1. Application/news-checkgroups .......................... 78 7.5. Cancel .................................................... 78 7.6. Ihave, sendme ............................................. 80 7.7. Obsolete control messages. ............................... 81 8. Duties of Various Agents ...................................... 81 8.1. General principles to be followed ......................... 81 8.2. Duties of an Injecting Agent .............................. 82 8.2.1. Proto-articles ........................................ 82 8.2.2. Procedure to be followed by Injecting Agents .......... 82 8.3. Duties of a Relaying Agent ................................ 84 8.4. Duties of a Serving Agent ................................. 85 8.5. Duties of a Posting Agent ................................. 86 8.6. Duties of a Followup Agent ................................ 86 8.7. Duties of a Moderator ..................................... 87 8.8. Duties of a Gateway ....................................... 88 8.8.1. Duties of an Outgoing Gateway ......................... 89 8.8.2. Duties of an Incoming Gateway ......................... 90 8.8.3. Example ............................................... 92 9. Security and Related Considerations ........................... 93 9.1. Leakage ................................................... 93 9.2. Attacks ................................................... 93 9.2.1. Denial of Service ..................................... 93 9.2.2. Compromise of System Integrity ........................ 94 9.3. Liability ................................................. 95 10. References ................................................... 96 11. Acknowledgements ............................................. 98 12. Contact Addresses ............................................ 98 13. Intellectual Property Rights ................................. 99 Appendix A.1 - A-News Article Format .............................. 100 Appendix A.2 - Early B-News Article Format ........................ 100 Appendix A.3 - Obsolete Headers ................................... 101 Appendix A.4 - Obsolete Control Messages .......................... 101 Appendix B - Collected Syntax ..................................... 102 C. H. Lindsey [Page 4] News Article Format July 2001 1. Introduction 1.1. Basic Concepts "Netnews" is a set of protocols for generating, storing and retrieving news "articles" (which resemble mail messages) and for exchanging them amongst a readership which is potentially widely distributed. It is organized around "newsgroups", with the expectation that each reader will be able to see all articles posted to each newsgroup in which he participates. These protocols most commonly use a flooding algorithm which propagates copies throughout a network of participating servers. Typically, only one copy is stored per server, and each server makes it available on demand to readers able to access that server. An important characteristic of Netnews is the lack of any requirement for a central administration or for the establishment of any controlling host to manage the network. A network which limits participation to some restricted set of hosts (within some company, for example) is a "closed" network; otherwise it is an "open" network. A set of hosts within a network which, by mutual arrangement, operates some variant (whether more or less restrictive) of the Netnews protocols is a "cooperating subnet". "Usenet" is a particular worldwide open network based upon the Netnews protocols, with the newsgroups being organised into recognized "hierarchies". Anybody can join (it is simply necessary to negotiate an exchange of articles with one or more other participating hosts). Usenet "belongs" to those who administer the hosts of which it is comprised. There is no Cabal with overall authority to direct what is to be be allowed. Nevertheless, there do exist agencies within Usenet that have authority to establish policies and to perform administrative functions, but such authority derives solely from the consent of those sites which choose to recognise it (and who can decline to exchange articles with sites which choose not to recognise it). Usually, the authority of such an agency is restricted to a particular hierarchy, or group of hierarchies. A "policy" is a rule intended to facilitate the smooth operation of a network by establishing parameters which restrict behaviour that, whilst technically unexceptionable, would nevertheless contravene some accepted standard of "Good Netkeeping". Since the ultimate beneficiaries of a network are its human readers, who will be less tolerant of poorly designed interfaces than mere computers, articles in breach of established policy can cause considerable annoyance to their recipients. Policies may well vary from network to network, from hierarchy to hierarchy within one network, and even between individual newsgroups within one hierarchy. It is assumed, for the purposes of this standard, that agencies with varying degrees of authority to establish such policies will exist, and that where they do not, policy will be established by mutual agreement. For the benefit of C. H. Lindsey [Page 5] News Article Format July 2001 networks and hierarchies without such established agencies, and to provide a basis upon which all agencies can build, this present standard often provides default policy parameters, usually introducing them by a phrase such as "As a matter of policy ...". 1.2. Objectives The purpose of this present standard is to define the protocols to be used for Netnews in general, and for Usenet in particular, and to set standards to be followed by software that implements those protocols. It is NOT the purpose of this standard to define how the authority of various agencies to exercise control or oversight of the various parts of Usenet is established (that is itself a matter of policy). Nevertheless, it is assumed that such authorities will exist, and tools are provided within the protocols for their use. 1.3. Historical Outline Network news originated as the medium of communication for Usenet, circa 1980. Since then, Usenet has grown explosively, and many Internet and non-Internet sites participate in it. In addition, the news technology is now in widespread use for other purposes, on the Internet and elsewhere. The earliest news interchange used the so-called "A News" article format. Shortly thereafter, an article format vaguely resembling Internet mail was devised and used briefly. Both of those formats are completely obsolete; they are documented in Appendix A.1 and Appendix A.2 for historical reasons only. With publication of [RFC 850] in 1983, news articles came to closely resemble Internet mail messages, with some restrictions and some additional headers. [RFC 1036] in 1987 updated [RFC 850] without making major changes. A Draft popularly referred to as "Son of 1036" [Son-of-1036] was written in 1994 by Henry Spencer. That document formed the original basis for this standard. Much is taken directly from Son of 1036, and it is hoped that we have followed its spirit and intentions. 1.4. Transport As in this standard's predecessors, the exact means used to transmit articles from one host to another is not specified. NNTP [NNTP] is the most common transmission method on the Internet, but much transmission takes place entirely independent of the Internet. Other methods in use include the UUCP protocol [RFC 976] extensively used in the early days of Usenet, FTP, downloading via satellite, tape archives, and physically delivered magnetic and optical media. 2. Definitions, Notations and Conventions C. H. Lindsey [Page 6] News Article Format July 2001 2.1. Definitions An "article" is the unit of news, analogous to an [RFC 2822] "message". A "proto-article" is one that has not yet been injected into the news system. A "message identifier" (5.3) is a unique identifier for an article, usually supplied by the "posting agent" which posted it or, failing that, by the "injecting agent". It distinguishes the article from every other article ever posted anywhere. Articles with the same message identifier are treated as if they are the same article regardless of any differences in the body or headers. A "newsgroup" is a single news forum, a logical bulletin board, having a name and nominally intended for articles on a specific topic. An article is "posted to" a single newsgroup or several newsgroups. When an article is posted to more than one newsgroup, it is said to be "crossposted"; note that this differs from posting the same text as part of each of several articles, one per newsgroup. A newsgroup may be "moderated", in which case submissions are not posted directly, but mailed to a "moderator" for consideration and possible posting. Moderators are typically human but may be implemented partially or entirely in software. A "hierarchy" is the set of all newsgroups whose names share a first component (as defined in 5.5). The term "sub-hierarchy" is also used where several initial components are shared. A "poster" is the person or software that composes and submits a possibly compliant article to a "posting agent". The poster is analogous to [RFC 2822]'s author(s). A "posting agent" is the software that assists posters to prepare proto-articles, in compliance with this standard. The proto-article is then passed on to an "injecting agent" for final checking and injection into the news stream. If the article is not compliant, or is rejected by the injecting agent, then the posting agent informs the poster with an explanation of the error. A "reader" is the person or software reading news articles. A "reading agent" is software which presents articles to a reader. A "followup" is an article containing a response to the contents of an earlier article (the followup's "precursor"). A "followup agent" is a combination of reading agent and posting agent that aids in the preparation and posting of a followup. An article's "reply address" is the address to which mailed replies should be sent. This is the address specified in the article's From header (5.2), unless it also has a Reply-To header (6.1). C. H. Lindsey [Page 7] News Article Format July 2001 A "reply agent" is a combination of reading agent and mailer that aids in the preparation and posting of an email response to an article. A "sender" is the person or software (usually, but not always, the same as the poster) responsible for the operation of the posting agent or, which amounts to the same thing, for passing the article to the injecting agent. The sender is analogous to [RFC 2822]'s sender. An "injecting agent" takes the finished article from the posting agent (often via the NNTP "post" command) performs some final checks and passes it on to a relaying agent for general distribution. A "relaying agent" is software which receives allegedly compliant articles from injecting agents and/or other relaying agents, and possibly passes copies on to other relaying agents and serving agents. A "news database" is the set of articles and related structural information stored by a serving agent and made available for access by reading agents. A "serving agent" receives an article from a relaying agent and files it in a news database. It also provides an interface for reading agents to access the news database. A "control message" is an article which is marked as containing control information; a relaying or serving agent receiving such an article may (subject to the policies observed at that site) take actions beyond just filing and passing on the article. A "gateway" is software which receives news articles and converts them to messages of some other kind (e.g. mail to a mailing list), or vice versa; in essence it is a translating relaying agent that straddles boundaries between different methods of message exchange. The most common type of gateway connects newsgroup(s) to mailing list(s), either unidirectionally or bidirectionally, but there are also gateways between news networks using this standard's news format and those using other formats. 2.2. Textual Notations This standard contains explanatory NOTEs using the following format. These may be skipped by persons interested solely in the content of the specification. The purpose of the notes is to explain why choices were made, to place them in context, or to suggest possible implementation techniques. NOTE: While such explanatory notes may seem superfluous in principle, they often help the less-than-omniscient reader grasp the purpose of the specification and the constraints involved. Given the limitations of natural language for descriptive purposes, this improves the probability that implementors and users will understand the true intent of the specification in C. H. Lindsey [Page 8] News Article Format July 2001 cases where the wording is not entirely clear. "US-ASCII" is short for "the ANSI X3.4 character set" [ANSI X3.4]. While "ASCII" is often misused to refer to various character sets somewhat similar to X3.4, in this standard "US-ASCII" is used to mean X3.4 and only X3.4. US-ASCII is a 7 bit character set. Please note that this standard requires that all agents be 8 bit clean; that is, they must accept and transmit data without changing or omitting the 8th bit. Certain words, when capitalized, are used to define the significance of individual requirements. The key words "MUST", "REQUIRED", "SHOULD", "RECOMMENDED", "MAY" and "OPTIONAL", and any of those words associated with the word "NOT", are to be interpreted as described in [RFC 2119]. In addition, the word "Ought", when applied to a poster, or to actions of posting and similar agents which a poster may easily override, indicates a recommendation whose violation would do no more than breach established policy, or accepted best practice. NOTE: The use of "MUST" or "SHOULD" implies a requirement that would or could lead to interoperability problems if not followed. Although not following an "Ought" recommendation might do no worse than cause extreme irritation to other readers, particularly in the case of the publicly distributed Usenet, that is no reason not to take it seriously. The essential distinction is that enforcement of a "MUST" or "SHOULD" is a matter of ensuring correct implementation, whereas enforcement of an "Ought" is more a matter of sensible design or of social pressure (whose effectiveness should not be underestimated, even though it cannot be prescribed by this standard). NOTE: A requirement imposed on a relaying or serving agent should be understood as applying only to articles actually accepted for processing by that agent (since any agent may always reject any article entirely, for reasons of site policy). All numeric values are given in decimal unless otherwise indicated. Octets are assumed to be unsigned values for this purpose. Throughout this standard we will give examples of various definitions, headers and other specifications. It needs to be remembered that these samples are for the aid of the reader only and do NOT define any specification themselves. In order to prevent possible conflict with "Real World" entities and people the top level domain of ".example" is used in all sample domains and addresses. The hierarchy of example.* is also used as a sample hierarchy. Information on the ".example" top level domain is in [RFC 2606]. C. H. Lindsey [Page 9] News Article Format July 2001 2.3. Relation To Mail and MIME The primary intent of this standard is to describe the news article format. Insofar as news articles are a subset of the Mail message format augmented by some new headers, this standard incorporates many (though not all) of the provisions of [RFC 2822], with the aim of enabling news articles to pass through mail systems and vice versa, provided only that they contain the minimum headers required for the mode of transport being used. Unfortunately, the match is not perfect, but it is the intention of this standard that gateways between Mail and News should be able to operate with the minimum of tinkering. Likewise, this standard incorporates many (though not all) of the provisions of the MIME standards [RFC 2045] et seq which, though designed with Mail in mind, are mostly applicable to News. 2.4. Syntax Notation This standard uses the Augmented Backus Naur Form described in [RFC 2234]. A discussion of this is outside the bounds of this standard, but it is expected that implementors will be able quickly to understand it with reference to that defining document. Much of the syntax of News Articles is based on the corresponding syntax defined in [RFC 2822] or in the MIME specifications [RFC 2045] et seq, which is deemed to have been incorporated into this standard as required. However, there are some important differences arising from the fact that [RFC 2822] does not recognise anything other than US-ASCII characters, that it does not recognise the MIME headers [RFC 2045], and that it includes much syntax described as "obsolete". NOTE: News parsers historically have been much less permissive than Mail parsers, and this is reflected in the modifications referred to, and in some further specific rules. The following syntactic forms therefore supersede the corresponding rules given in [RFC 2822] and [RFC 2045], thus allowing UTF-8 characters [RFC 2279] to appear in certain contexts (the five rules begining with "strict-" reflect the corresponding original rules from [RFC 2822]). UTF8-xtra-2-head= %xC2-DF UTF8-xtra-3-head= %xE0 %xA0-BF / %xE1-EC %x80-BF / = %xED %x80-9F / %xEE-EF %x80-BF UTF8-xtra-4-head= %xF0 %x90-BF / %xF1-F7 %x80-BF UTF8-xtra-5-head= %xF8 %x88-BF / %xF9-FB %x80-BF UTF8-xtra-6-head= %xFC %x84-BF / %xFD %x80-BF UTF8-xtra-tail = %x80-BF UTF8-xtra-char = UTF8-xtra-2-head 1( UTF8-xtra-tail ) / UTF8-xtra-3-head 1( UTF8-xtra-tail ) / UTF8-xtra-4-head 2( UTF8-xtra-tail ) / UTF8-xtra-5-head 3( UTF8-xtra-tail ) / UTF8-xtra-6-head 4( UTF8-xtra-tail ) C. H. Lindsey [Page 10] News Article Format July 2001 text = %d1-9 / ; all UTF-8 characters except %d11-12 / ; US-ASCII NUL, CR and LF %d14-127 / UTF8-xtra-char ctext = NO-WS-CTL / ; all of except %d33-39 / ; SP, HTAB, "(", ")" %d42-91 / ; and "\" %d93-126 / UTF8-xtra-char qtext = NO-WS-CTL / ; all of except %d33 / ; SP, HTAB, "\" and DQUOTE %d35-91 / %d93-126 / UTF8-xtra-char utext = NO-WS-CTL / ; Non white space controls %d33-126 / ; The rest of US-ASCII UTF8-xtra-char strict-text = %d1-9 / ; text restricted to %d11-12 / ; US-ASCII %d14-127 strict-qtext = NO-WS-CTL / ; qtext restricted to %d33 / ; US-ASCII %d35-91 / %d93-127 strict-quoted-pair = "\" strict-text strict-qcontent = strict-qtext / strict-quoted-pair strict-quoted-string = [CFWS] DQUOTE *([FWS] strict-qcontent) [FWS] DQUOTE [CFWS] The syntax for UTF8-xtra-char excludes those redundant sequences of octets which cannot occur in UTF-8, as defined by [RFC 2279], either because they would not be the shortest possible encodings of some UCS character, or they would represent one of the characters D800 through DFFF, disallowed in UCS because of their surrogate use in the UTF-16 encoding. These sequences MUST NOT be generated by posting agents. Where they occur inadavertently, they MAY be passed on untouched by other agents, but they MUST NOT ever be interpreted as valid characters. Wherever in this standard the syntax is stated to be taken from [RFC 2822], it is to be understood as the syntax defined by [RFC 2822] after making the above changes, but NOT including any syntax defined in section 4 ("Obsolete syntax") of [RFC 2822]. Software compliant with this standard MUST NOT generate any of the syntactic forms defined in that Obsolete Syntax, although it MAY accept such syntactic forms. Certain syntax from the MIME specifications [RFC 2045] et seq is also considered a part of this standard (see 6.21). The following syntactic forms, taken from [RFC 2234] or from [RFC 2822], are repeated here for convenience only: C. H. Lindsey [Page 11] News Article Format July 2001 ALPHA = %x41-5A / ; A-Z %x61-7A ; a-z CR = %x0D ; carriage return CRLF = CR LF DIGIT = %x30-39 ; 0-9 HTAB = %x09 ; horizontal tab LF = %x0A ; line feed SP = %x20 ; space NO-WS-CTL = %d1-8 / ; US-ASCII control characters %d11 / ; which do not include the %d12 / ; carriage return, line feed, %d14-31 / ; and whitespace characters %d127 specials = "(" / ")" / ; Special characters used in "<" / ">" / ; other parts of the syntax "[" / "]" / ":" / ";" / "@" / " "," / "." / DQUOTE WSP = SP / HTAB ; Whitespace characters FWS = ([*WSP CRLF] 1*WSP); Folding whitespace ccontent = ctext / quoted-pair / comment comment = "(" *([FWS] ccontent) [FWS] ")" CFWS = *([FWS] comment) (([FWS] comment) / FWS ) DQUOTE = %d34 ; quote mark quoted-pair = "\" text atext = ALPHA / DIGIT / "!" / "#" / ; Any character except "$" / "%" / ; controls, SP, and specials. "&" / "'" / ; Used for atoms "*" / "+" / "-" / "/" / "=" / "?" / "^" / "_" / "`" / "}" / "|" / "}" / "~" atom = [CFWS] 1*atext [CFWS] dot-atom = [CFWS] dot-atom-text [CFWS] dot-atom-text = 1*atext *( "." 1*atext ) qcontent = qtext / quoted-pair quoted-string = [CFWS] DQUOTE *([FWS] qcontent) [FWS] DQUOTE [CFWS] word = atom / quoted-string phrase = 1*word unstructured = *( [FWS] utext ) [FWS] NOTE: CFWS occurs at many places in the syntax in order to allow comments and extra whitespace to be inserted almost anywhere. The syntax is in fact ambiguous insofar as it may be impossible to tell in which of several possible ways a given comment or WS was produced. However, this does not lead to semantic ambiguity C. H. Lindsey [Page 12] News Article Format July 2001 because, unless specifically stated otherwise, the presence of absence of a comment or additional WS has no semantic meaning and, in particular, it is a matter of indifference whether it forms a part of the syntactic construct preceding it or the one following it. NOTE: Following [RFC 2234], literal text included in the syntax is to be regarded as case-insensitive. However, in contradistinction to [RFC 2822], the Netnews protocols are sensitive to case in some instances (as in newsgroup names, some header parameters, etc.). Care has been taken to indicate this explicitly where required. The complete syntax defined in this standard is repeated, for convenience, in Appendix B. 2.5. Language Various constant strings in this standard, such as header names and month names, are derived from English words. Despite their derivation, these words do NOT change when the poster or reader employing them is interacting in a language other than English. Posting and reading agents MAY translate as appropriate in their interaction with the poster or reader, but the forms that actually appear in articles MUST be the English-derived ones defined in this standard. 3. Changes to the existing protocols This standard prescribes many changes, clarifications and new features since the protocols described in [RFC 1036] and [Son-of- 1036]. It is the intention that they can be assimilated into Usenet as it presently operates without major interruption to the service, though some of the new features may not begin to show benefit until they become widely implemented. This section summarizes the main changes, and comments on some features of the transition. 3.1. Principal Changes o The [RFC 2822] conventions for parenthesis-enclosed comments in headers are supported. o Whitespace is permitted in Newsgroups headers, permitting folding of such headers. Indeed, all news headers can now be folded. o An enhanced syntax for the Path header enables the injection point of and the route taken by an article to be determined with certainty. o Netnews is firmly established as an 8bit medium. o Large parts of MIME are recognised as an integral part of Netnews. o The charset for headers is always UTF-8. This will, inter alia, permit newsgroup-names with non-ASCII characters. o There is a new Control command 'mvgroup' to facilitate moving a group to a different place (name) in a hierarchy. o There are several new headers defined, such as Replaces and C. H. Lindsey [Page 13] News Article Format July 2001 Author-Ids, leading to increased functionality. o There are numerous other small changes, clarifications and enhancements. [Doubtless many other changes should be listed, but there is little point in doing so until our text is nearing completion. The above gives the flavour of what should be said. There should also be references to Appendix A.3 and Appendix A.4 ] 3.2. Transitional Arrangements An important distinction must be made between serving and relaying agents which are responsible for the distribution and storage of news articles, and user agents which are responsible for interactions with users. It is important that the former should be upgraded to conform to this standard as soon as possible to provide the benefit of the enhanced facilities. Fortunately, the number of distinct implementations of such agents is rather small, at least so far as the main "backbone" of Usenet is concerned, and many of the new features are already supported. Contrariwise, there are a great number of implementations of user agents, installed on a vastly greater number of small sites. Therefore, the new functionality has been designed so that existing agents may continue to be used, although the full benefits may not be realised until a substantial proportion of them have been upgraded. In the list which follows, care has been taken to distinguish the implications for both kinds of agent. o [RFC 2822] style comments in headers do not affect serving and relaying agents (note that the Newsgroups and Path headers do not contain them). They are unlikely to hinder their proper display in existing user agents except in the case of the References header in agents which thread articles. Therefore, it is provided that they SHOULD NOT be generated except where permitted by the previous standards. o Because of its importance to all serving agents, the extension permitting whitespace and folding in Newsgroup headers SHOULD NOT be used until it has been widely deployed amongst relaying agents. User agents are unaffected. o The new style of Path header is already consistent with the previous standards. However, the intention is that relaying agents should eventually reject articles in the old style, and so this should be offered as a configurable option for relaying agents. User agents are unaffected. o The vast majority of serving, relaying and transport agents are believed to be already 8bit clean (in the slightly restricted sense in which that term is used in the MIME standards). User agents that do not implement MIME may be disadvantaged, but no more so than at present when faced with 8bit characters (which currently abound in spite of the previous standards). o The introduction of MIME reflects a practice that is already widespread. Articles in strict compliance with the previous standards (using strict US-ASCII) will be unaffected. Many user agents already support it, at least to the extent of widely used C. H. Lindsey [Page 14] News Article Format July 2001 charsets such as ISO-8859-1. Users expecting to read articles using other charsets will need to acquire suitable reading agents. It is not intended, in general, that any single user agent will be able to display every charset known to IANA, but all such agents MUST support US-ASCII. Serving and relaying agents are not affected. o The use of the UTF-8 charset for headers will not affect any existing usage, since US-ASCII is a strict subset of UTF-8. Insofar as newsgroup names containing non-ASCII characters can now be expected to arise, support from serving and relaying agents will be necessary. It is believed that the customary storage structure used by serving agents can already cope (perhaps not ideally) with such names. Note that it is not necessary for serving and relaying agents to understand all the characters available in UTF-8, though it is desirable for them to be displayable for diagnostic purposes via some escape mechanism using, for example, the visible subset of US-ASCII. For users expecting to use the more exotic possibilities available under UTF-8, the remarks already made in connection with MIME will apply. o The new Control: mvgroup command will need to be implemented in serving agents. It SHOULD be used in conjunction with pairs of matching rmgroup and newgroup commands (injected shortly after the mvgroup) until such time as mvgroup is widely implemented. The new Replaces header is also effectively a Control command, and transitional arrangements are provided which should be used in the meantime. User agents are unaffected. o The headers newly introduced by this standard can safely be ignored by existing software, albeit with loss of the new functionality. 4. Basic Format 4.1. Syntax of News Articles The overall syntax of a news article is: article = 1*header separator body header = header-name ":" 1*SP header-content CRLF header-name = 1*name-character *( "-" 1*name-character ) name-character = ALPHA / DIGIT header-content = USENET-header-content *( [CFWS] ";" header-parameter ) / other-header-content USENET-header-content = other-header-content = header-parameter = USENET-header-parameter / other-header-parameter C. H. Lindsey [Page 15] News Article Format July 2001 USENET-header-parameter = other-header-parameter = attribute "=" value attribute = USENET-token / iana-token / x-token value = token / quoted-string USENET-token = iana-token = x-token = [CFWS] "x-" token-core [CFWS] token = [CFWS] token-core [CFWS] token-core = 1* tspecials = "(" / ")" / "<" / ">" / "@" / "," / ";" / ":" / "\" / DQUOTE / "/" / "[" / "]" / "?" / "=" separator = CRLF body = *( *998text CRLF ) An article consists of some headers followed by a body. An empty line separates the two. The headers contain structured information about the article and its transmission. A header begins with a header-name identifying it, and can be continued onto subsequent lines as described in section 4.2.3. The body is largely unstructured text significant only to the poster and the readers. NOTE: Terminology here follows the current custom in the news community, rather than the [RFC 2822] convention of referring to what is here called a "header" as a "header-field" or "field". Note that the separator line must be truly empty, not just a line containing white space. Further empty lines following it are part of the body, as are empty lines at the end of the article. NOTE: The syntax above defines the canonical form of a news article as a sequence of lines each terminated by CRLF. This does not prevent serving agents or transport agents from storing or handling the article in other formats (e.g. using a single LF in place of CRLF) so long as the overall effects achieved are as defined by this standard when operating on the canonical form. 4.2. Headers 4.2.1. Names and Contents Despite the restrictions on header-name syntax imposed by the grammar, relayers and reading agents SHOULD tolerate header names containing any US-ASCII printable character other than colon (":", US-ASCII 58). C. H. Lindsey [Page 16] News Article Format July 2001 Header-names SHOULD be either those for which a USENET-header-content is established by this standard, or by [RFC 2822], or by any extension to either of these standards including, in particular, the MIME standards [RFC 2045] et seq., or else experimental headers beginning with "X-" (as defined in 4.2.2.1). Software SHOULD NOT attempt to interpret headers not described in this standard or in its extensions, but relaying agents MUST pass them on unaltered and reading agents MUST enable them to be displayed, at least optionally. The possibility of allowing header-parameters to appear in all headers is provided mainly for the purpose of allowing future extensions to existing headers, since only a very few USENET-header- parameters are actually defined in this standard. Observe that such header-parameters do not, in general, occur in headers defined in other standards, except for the MIME standards [RFC 2045] et seq. and their extensions. Nevertheless, compliant software MUST accept all such header-parameters in headers defined by this standard and its extensions (ignoring them if their meaning is unknown) and SHOULD accept (and ignore) them in all headers. [but what about address = mailbox / group group = phrase ":" [mailbox-list] ";" Does the following NOTE cover the situation?] NOTE: The presence of a ";" in a header-content does not indicate the presence of a header-parameter in the few situations where it can be parsed as part of some USENET- header-content or other-header-content. On the other hand, posting agents SHOULD NOT generate header- parameters (even those using x-tokens) except in those headers for which a USENET-header-parameter has been defined, or where that usage is permitted by some other standard (notably one of the MIME standards). This restriction is likely to removed in a future version of this standard. NOTE: The given syntax is ambiguous insofar as a USENET-header- content that is defined to be could contain, within that , text of the form <*(";" header- parameter)>. The intention is therefore that any such apparent header-parameters are to be regarded as part of the . This standard therefore does not (and extensions to it SHOULD NOT) define any USENET-header-parameter to be associated with such an unstructured USENET-header-content. The order of headers in an article is not significant. However, posting agents are encouraged to put mandatory headers (section 5) first, followed by optional headers (section 6), followed by experimental headers and headers not defined in this standard or its extensions. Relaying agents MUST NOT change the order of the headers in an article. C. H. Lindsey [Page 17] News Article Format July 2001 Header-names are case-insensitive. There is a preferred case convention, which posters and posting agents Ought to use: each hyphen-separated "word" has its initial letter (if any) in uppercase and the rest in lowercase, except that some abbreviations have all letters uppercase (e.g. "Message-ID" and "MIME-Version"). The forms used in this standard are the preferred forms for the headers described herein. Relaying and reading agents MUST, however, tolerate articles not obeying this convention. 4.2.2. Header Properties There are three special properties that may apply to particular headers, namely: "experimental", "inheritable", and "variant". When a header is defined, in this (or any future) standard, as having one (or possibly more) of these properties, it is subject to special treatment, as indicated below. 4.2.2.1. Experimental Headers Experimental headers are those whose header-names begin with "X-". They are to be used for experimental Netnews features, or for enabling additional material to be propagated with an article. There are no established headers (see 4.2.1) that are considered experimental headers; an established header cannot be experimental. NOTE: Some such headers may eventually be adopted as standard by some extension to this standard, at which point they will lose their "X-" prefix. 4.2.2.2. Inheritable Headers Subject only to the overriding ability of the poster to determine the contents of the headers in a proto-article, headers with the inheritable property MUST be copied by followup agents (perhaps with some modification) into the followup article, and headers without that property MUST NOT be so copied. Examples include: o Newsgroups (5.5) - copied from the precursor, subject to any Followup-To header. o Subject (5.4) - modified by prefixing with "Re: ", but otherwise copied from the precursor. o References (6.10) - copied from the precursor, with the addition of the precursor's Message-ID. o Distribution (6.6) - copied from the precursor. NOTE: The Keywords header is not inheritable, though some older newsreaders treated it as such. 4.2.2.3. Variant Headers Headers with the variant property may differ between (or even be completely absent from) copies of the same article as stored or relayed throughout a Netnews system. The manner of the difference (or absence) MUST be as specified in this (or any future) standard. Typically, these headers are modified as articles are propagated, or C. H. Lindsey [Page 18] News Article Format July 2001 they reflect the status of the article on a particular serving agent, or cooperating group of such agents. The variant header MAY be placed anywhere within the headers (though placing it first is recommended). The principle examples are: o Path (5.6) - augmented at each relaying agent that an article passes through. o Xref (6.16) - used to keep track of the article locators of crossposted articles so that newsreaders serviced by a particular serving agent can mark such articles as read. 4.2.3. White Space and Continuations Each header is logically a single line of characters comprising the header-name, the colon with its following SP, and the header-content. For convenience, however, the header-content can be split into a multiple line representation; this is called "folding". The general rule is that wherever this standard allows for FWS or CFWS (but not simply SP or HTAB) a CRLF may be inserted before any WSP. For example, the header: Approved: modname@modsite.example (Moderator of comp.foo.bar) can be represented as: Approved: modname@modsite.example (Moderator of comp.foo.bar) NOTE: Though header-contents are defined in such a way that folding can take place between many of the lexical tokens (and even within some of them), folding SHOULD be limited to placing the CRLF at higher-level syntactic breaks, and SHOULD also avoid leaving trailing WSP on the preceding line. For instance, if a header-content is defined as comma-separated values, it is recommended that folding occur after the comma separating the structured items, even if it is allowed elsewhere. Folding MUST NOT be carried out in such a way that any line of a header is made up entirely of WSP characters and nothing else. The colon following the header name on the first line MUST be followed by a WSP, even if the header is empty. If the header is not empty, at least some of the content MUST appear on the first line (this is to avoid the possibility of harm by any non-compliant agent that might eliminate a trailing SP). Posting agents MUST enforce these restrictions, but relaying agents SHOULD accept even articles that violate them. NOTE: This standard differs from [RFC 2822] in requiring that WSP followng the colon (it was also an [RFC 1036] requirement). Posters and posting agents SHOULD use SP, not HTAB, where white space is desired in headers (some existing software expects this), and MUST use SP immediately following the colon after a header-name. Relaying agents SHOULD accept HTAB in all such cases, however. C. H. Lindsey [Page 19] News Article Format July 2001 Since the white space beginning a continuation line remains a part of the logical line, headers can be "broken" into multiple lines only at FWS or CFWS. Posting agents Ought Not to break headers unnecessarily (but see 4.5). 4.2.4. Comments Strings of characters which are treated as comments may be included in header-contents wherever the syntactic element CFWS occurs. They consist of characters enclosed in parentheses. Such strings are considered comments so long as they do not appear within a quoted- string. Comments may be nested. A comment is normally used to provide some human readable informational text, except at the end of an address which contains no phrase, as in fred@foo.bar.example (Fred Bloggs) as opposed to "Fred Bloggs" . The former is a deprecated, but commonly encountered, usage and reading agents SHOULD take special note of such comments as indicating the name of the person whose address it is. In all other situations a comment is semantically interpreted as a single SP. Since a comment is allowed to contain FWS, folding is permitted within it as well as immediately preceding and immediately following it. Also note that, since quoted-pair is allowed in a comment, the parenthesis and backslash characters may appear in a comment so long as they appear as a quoted-pair. Semantically, the enclosing parentheses are not part of the comment content; the content is what is contained between the two parentheses. Since comments have not hitherto been permitted in news articles, except in a few specified places, posters and posting-agents SHOULD NOT insert them except in those places, namely following addresses in From and similar headers, and to indicate the name of the timezone in Date headers. However, compliant software MUST accept them in all places where they are syntactically allowed. 4.2.5. Undesirable Headers A header whose content is empty is said to be an empty header. Relaying and reading agents SHOULD NOT consider presence or absence of an empty header to alter the semantics of an article (although syntactic rules, such as requirements that certain header names appear at most once in an article, MUST still be satisfied). Posting and injecting agents SHOULD delete empty headers from articles before posting them; relaying agents MUST pass them untouched. Headers that merely state defaults explicitly (e.g., a Followup-To header with the same content as the Newsgroups header, or a MIME Content-Type header with contents "text/plain; charset=us-ascii") or state information that reading agents can typically determine easily themselves (e.g. the length of the body in octets) are redundant and C. H. Lindsey [Page 20] News Article Format July 2001 posters and posting agents Ought Not to include them. 4.3. Body 4.3.1. Body Format Issues The body of an article SHOULD NOT be empty. A posting or injecting agent which does not reject such an article entirely SHOULD at least issue a warning message to the poster and supply a non-empty body. Note that the separator line MUST be present even if the body is empty. NOTE: Some existing news software is known to react badly to body-less articles, hence the request for posting and injecting agents to insert a body in such cases. The sentence "This article was probably generated by a buggy news reader" has traditionally been used is this situation. Note that an article body is a sequence of lines terminated by CRLFs, not arbitrary binary data, and in particular it MUST end with a CRLF. However, relaying agents SHOULD treat the body of an article as an uninterpreted sequence of octets (except as mandated by changes of CRLF representation and by control-message processing) and SHOULD avoid imposing constraints on it. See also section 4.5. Posters SHOULD avoid using control characters and escape sequences except for tab (US-ASCII 9), formfeed (US-ASCII 12) and, possibly, backspace (US-ASCII 8). Tab signifies sufficient horizontal white space to reach the next of a set of fixed positions; posters are warned that there is no standard set of positions, so tabs should be avoided if precise spacing is essential. Formfeed (which is sometimes referred to as the "spoiler character") signifies a point at which a reading agent Ought to pause and await reader interaction before displaying further text. NOTE: Passing other control characters or escape sequences unaltered to a display or printing device is likely to have unpredictable results, except in the case of a device adapted to the special needs of some particular character set. NOTE: Backspace was historically used for underlining, done by an underscore (US-ASCII 95), a backspace, and a character, repeated for each character that should be underlined. Posters are warned that underlining is not available on all output devices or supported by all reading agents and is best not relied on for essential meaning. 4.3.2. Body Conventions A body is by default an uninterpreted sequence of octets for most of the purposes of this standard. However, a MIME Content-Type header may impose some structure or intended interpretation upon it, and may also specify the character set in accordance with which the octets are to be interpreted. C. H. Lindsey [Page 21] News Article Format July 2001 It is a common practice for followup agents to enable the incorporation of the followed-up article (the "precursor") as a quotation. This SHOULD be done by prefacing each line of the quoted text (even if it is empty) with the character ">" (or perhaps with "> " in the case of a previously unquoted line). This will result in multiple levels of ">" when quoted content itself contains quoted content, and it will also facilitate the automatic analysis of articles. NOTE: Posters should edit quoted context to trim it down to the minimum necessary. However, followup agents Ought Not to attempt to enforce this beyond issuing a warning (past attempts to do so have been found to be notably counter-productive). The followup agent SHOULD also precede the quoted content by an "attribution line" (however, readers are warned not to assume that they are accurate, especially within multiply nested quotations). The following convention for such lines, whilst not mandated by this standard, is intended to facilitate their automatic recognition and processing by sophisticated reading agents. The attribution SHOULD contain the name or the email address of the precursor's poster, as in Joe D. Bloggs wrote: or Helmut Schmidt schrieb: The attribution MAY contain also a single Newsgroup name (the one from which the followup is being made), the precursor's Message-ID and/or the precursor's Date and Time. Any of these that are present, SHOULD precede the name and/or email address. However, the inclusion or not of such fields Ought always to be under the control of the poster. To enable this line, and the Message-ID and the Email address within it, to be recognised (for example to enable suitable reading agents to retrieve the precursor or email its poster by clicking on them), the following conventions SHOULD be observed: o The precursor's Message-ID SHOULD be enclosed within <...> or o The precursor's poster's Email address SHOULD be enclosed within <...> o The various fields may be separated by arbitrary text and they may be folded in the same way as headers, but attributions SHOULD always be terminated by a ":" followed by CRLF. Further examples: On comp.foo in <1234@bar.example> on 24 Dec 1997 16:40:20 +0000, Joe D. Bloggs wrote: Am 24. Dez 1997 schrieb Helmut Schmidt : C. H. Lindsey [Page 22] News Article Format July 2001 A "personal signature" is a short closing text automatically added to the end of articles by posting agents, identifying the poster and giving his network addresses, etc. If a poster or posting agent does append such a signature to an article, it MUST be preceded with a delimiter line containing (only) two hyphens (US-ASCII 45) followed by one SP (US-ASCII 32). The signature is considered to extend from the last occurrence of that delimiter up to the end of the article (or up to the end of the part in the case of a multipart MIME body). Followup agents, when incorporating quoted text from a precursor, Ought Not to include the signature in the quotation. Posting agents Ought to discourage (at least with a warning) signatures of excessive length (4 lines is a commonly accepted limit). 4.4. Characters and Character Sets Transmission paths for news articles MUST treat news articles as uninterpreted sequences of octets, excluding the values 0 (US-ASCII NUL) and 13 and 10 (US-ASCII CR and LF, which MUST ONLY appear in the combination CRLF which denotes a line separator). NOTE: this correspponds to the range of octets permitted for MIME "8bit data" [RFC 2045]. Thus raw binary data cannot be transmitted in an article body except by the use of a Content- Transfer-Encoding such as base64. Character data is represented by octets in accordance with some encoding scheme (UTF-8 for headers, and determined by the Content- Type and Content-Transfer-Encoding headers for bodies). If it comes to a relaying agent's attention that it is being asked to pass an article using the Content-Transfer-Encoding "8bit" to a relaying agent that does not support it, it SHOULD report this error to its administrator. It MUST refuse to pass the article and MUST NOT re-encode it with different MIME encodings. NOTE: This strategy will do little harm. The target relaying agent is unlikely to be able to make use of the article on its own servers, and the usual flooding algorithm will likely find some alternative route to get the article to destinations where it is needed. 4.4.1. Character Sets within Article Headers Within article headers, characters are represented as octets according to the UTF-8 encoding scheme [RFC 2279] or [ISO/IEC 10646], and hence all the characters in Unicode [UNICODE 3.1] or in the Universal Multiple-Octet Coded Character Set (UCS) [ISO/IEC 10646] (which is essentially a superset of Unicode and expected to remain so) are potentially available. However, processing all octets in the same manner as US-ASCII characters should ensure correct behaviour in most situations. C. H. Lindsey [Page 23] News Article Format July 2001 NOTE: UTF-8 is an encoding for 16bit (and even 32bit) character sets with the property that any octet less than 128 immediately represents the corresponding US-ASCII character, thus ensuring upwards compatibility with previous practice. Non-ASCII characters from Unicode are represented by sequences of octets satisfying the syntax of a UTF8-xtra-char (2.4), which excludes certain octet sequences not explicitly permitted by [RFC 2279]. Unicode includes all characters from the ISO-8859 series of characters sets [ISO 8859] (which includes all Cyrillic, Greek and Arabic characters) together with the more elaborate characters used in Asian countries. See the following section for the appropriate treatment of Unicode characters by reading agents. Notwithstanding the great flexibility permitted by UTF-8, there is need for restraint in its use in order that the essential components of headers may be discerned using reading agents that cannot present the full Unicode range. In particular, header-names and tokens MUST be in US-ASCII, and certain other components of headers, as defined elsewhere in this standard - notably msg-ids, date-times, dot-atoms, domains and path-identities - MUST be in US-ASCII. Comments, phrases (as in addresses) and unstructureds (as in Subject headers) MAY use the full range of UTF-8 characters, but SHOULD nevertheless be invariant under Unicode normalization NFC [UNICODE 3.1]. NOTE: The effect of normalization NFC is to place composite characters (made by overlaying one character with another) into a canonical form (usually represented by a single character where one is available - thus E-acute is preferred over E followed by a non-spacing acute accent), and to make a consistent choice among equivalent forms (e.g. the Angstrom sign is replaced by A-ring). At least for the main European languages, for which all the needed composites are already available as single characters, it is unlikely that posting agents will need to take any special steps to ensure normalization. In the particular case of newsgroup-names (see 5.5) there are more stringent requirements regarding the use of UTF-8 and Unicode. Where the use of non-ASCII characters, encoded in UTF-8, is permitted as above, they MAY also be encoded using the MIME mechanism defined in [RFC 2047], but this usage is deprecated within news articles (even though it is required in mail messages) since it is less legible in older reading agents which support neither it nor UTF-8. Nevertheless, reading agents SHOULD support this usage, but only in those contexts explicitly mentioned in [RFC 2047]. 4.4.2. Character Sets within Article Bodies Within article bodies, characters are represented as octets according to the encoding scheme implied by any Content-Transfer-Encoding and Content-Type headers [RFC 2045]. In the absence of such headers, reading agents cannot be relied upon to display correctly more than C. H. Lindsey [Page 24] News Article Format July 2001 the US-ASCII characters. NOTE: Observe that reading agents are not forbidden to "guess", or to interpret as UTF-8 regardless, which would be the simplest course for them to take. NOTE: It is not expected that reading agents will necessarily be able to present characters in all possible character sets, although they MUST be able to present all US-ASCII characters. For example, a reading agent might be able to present only the ISO-8859-1 (Latin 1) characters [ISO 8859], in which case it Ought to present undisplayable characters using some distinctive glyph, or by exhibiting a suitable warning. Older reading agents that do not understand MIME headers or UTF-8 should be able to display bodies in US-ASCII (with some loss of human comprehensibility) except possibly when the Content-Transfer- Encoding is "8bit". Followup agents MUST be careful to apply appropriate encodings to the outbound followup. A followup to an article containing non-ASCII material is very likely to contain non-ASCII material itself. 4.5. Size Limits Posting agents SHOULD endeavour to keep all header lines, so far as is possible, within 79 characters by folding them at suitable places (see 4.2.3). However, posting agents MUST permit the poster to include longer headers if he so insists, and compliant software MUST support headers of at least 998 octets. Likewise, injecting agents SHOULD fold any headers generated automatically by themselves. Relaying agents MUST NOT fold headers (i.e. they must pass on the folding as received). NOTE: There is NO restriction on the number of lines into which a header may be split, and hence there is NO restriction on the total length of a header (in particular it may, by suitable folding, be made to exceed the 998 octets restriction pertaining to a single header line). The syntax provides for the lines of a body to be up to 998 octets in length, not including the CRLF. All software compliant with this standard MUST support lines of at least that length, both in headers and in bodies, and all such software SHOULD support lines of arbitrary length. In particular, relaying agents MUST transmit lines of arbitrary length without truncation or any other modification. NOTE: The limit of 998 octets is consistent with the corresponding limit in [RFC 2822]. In plain-text messages (those with no MIME headers, or those with a MIME Content-Type of text/plain) posting agents Ought to endeavour to keep the length of body lines within some reasonable limit. The size of this limit is a matter of policy, the default being to keep within 79 characters at most, and preferably within 72 characters (to allow C. H. Lindsey [Page 25] News Article Format July 2001 room for quoting in followups). Exceptionally, posting agents Ought Not to adjust the length of quoted lines in followups unless they are able to reformat them in a consistent manner. Moreover, posting agents MUST permit the poster to include longer lines if he so insists. NOTE: Plain-text messages are intended to be displayed "as-is" without any special action (such as automatic line splitting) on the part of the recipient. The policy limit (e.g. 72 or 79) should be expressed as a number of characters (as they will be displayed by a reading agent) rather than as the number of octets used to encode them. NOTE: This standard provides no upper bound on the overall size of a single article, but neither does it forbid relaying agents from dropping articles of excessive length. It is, however, suggested that any limits thought appropriate by particular agents would be more appropriately expressed in megabytes than in kilobytes. 4.6. Example Here is a sample article: Path: server.example/unknown.site2.example@site2.example/ relay.site.example/site.example/injector.site.example%jsmith Newsgroups: example.announce,example.chat Message-ID: <9urrt98y53@site.example> From: Ann Example Subject: Announcing a new sample article. Date: Fri, 27 Mar 1998 12:12:50 +1300 Approved: example.announce moderator Followup-To: example.chat Reply-To: Ann Example Expires: Wed, 22 Apr 1998 12:12:50 -0700 Organization: Site1, The Number one site for examples. User-Agent: ExampleNews/3.14 (Unix) Keywords: example, announcement, standards, RFC 1036, Usefor Summary: The URL for the next standard. Just a quick announcemnt that a new standard example article has been released; it is in the new USEFOR draft obtainable from ftp.ietf.org. Ann. -- Ann Example Sample Poster to the Stars "The opinions in this article are bloody good ones" - J. Clarke. C. H. Lindsey [Page 26] News Article Format July 2001 5. Mandatory Headers An article MUST have one, and only one, of each of the following headers: Date, From, Message-ID, Subject, Newsgroups, Path. Note also that there are situations, discussed in the relevant parts of section 6, where References, Sender, or Approved headers are mandatory. In control messages, specific values are required for certain headers. For the overall syntax of headers, see section 4.1. In the discussions of the individual headers, the content of each is specified using the syntax notation. The convention used is that the content of, for example, the Subject header is defined as . A proto-article (see 8.2.1) may lack some of these mandatory headers, but they MUST then be supplied by the injecting agent. 5.1. Date The Date header contains the date and time that the article was prepared by the poster ready for transmission and SHOULD express the poster's local time. The content syntax makes use of syntax defined in [RFC 2822], subject to the following revised definition of zone. Date-content = date-time zone = (( "+" / "-" ) 4DIGIT) / "UT" / "GMT" The forms "UT" and "GMT" (indicating universal time) are to be regarded as obsolete synonyms for "+0000". They MUST be be accepted, and passed on unchanged, by all agents, but they MUST NOT be generated as part of new articles by posting and injecting agents. Although folding white space is permitted throughout the date-time syntax, it is RECOMMENDED that a single space be used in each place that FWS appears (whether it is required or optional). NOTE: A convention that is sometimes followed is to add a comment, after the date-time, containing the time zone in human-readable form, but many of the abbreviations commonly used for this purpose are ambiguous. The value given by the is the only definitive form. In order to prevent the reinjection of expired articles into the news stream, relaying and serving agents MUST refuse articles whose Date header predates the earliest articles of which they normally keep record, or which is more than 24 hours into the future (though they MAY use a margin less than that 24 hours). Relaying agents MUST NOT modify the Date header in transit. C. H. Lindsey [Page 27] News Article Format July 2001 5.1.1. Examples Date: Fri, 2 Apr 1999 20:20:51 -0500 (EST) Date: 26 May 1999 16:13 +0000 Date: 26 May 1999 16:13 GMT (Obsolete) 5.2. From The From header contains the electronic address(es), and possibly the full name, of the article's poster(s). The content syntax makes use of syntax defined in [RFC 2822], subject to the following revised definition of local-part. From-content = mailbox-list addr-spec = local-part "@" domain local-part = dot-atom / strict-quoted-string NOTE: This syntax ensures that the local-part of an addr-spec is restricted to pure US-ASCII (and is thus in strict compliance with [RFC 2822]), whilst allowing any UTF-8 character to be used in a preceding quoted-string containing the poster's full name. If some future extension to the Mail protocols should relax this restriction, one would expect the Netnews protocols to follow. The mailbox in the From-content SHOULD be a valid address, belonging to the poster(s) of the article, or person or agent on whose behalf the post is being sent (see the Sender header, 6.2). When, for whatever reason, the poster does not wish to include such an adddress, the From-content SHOULD then be an address which ends in the top level domain of ".invalid" [RFC 2606]. NOTE: Since such addresses ending in ".invalid" are undeliverable, user agents Ought to warn any user attempting to reply to them and Ought Not, in any case, to attempt to deliver to them (since that would be pointless anyway). Whether or not a valid address can subsequently be extracted from such an address falls outside the scope of this standard (though it would be pointless to use a disguise so easily penetrable). Be warned also that some injecting agents that have authentication information may choose to replace the From- content based upon the authenticated identity. 5.2.1. Examples: From: John Smith From: "John Smith" , dave@isp.example From: "John D. Smith" , andrew@isp.example, fred@site2.example From: Jan Jones From: Jan Jones From: dave@isp.example (Dave Smith) C. H. Lindsey [Page 28] News Article Format July 2001 NOTE: the last example shows a now deprecated convention of putting a poster's full name in a comment following the mailbox, rather than in a phrase at the start of that mailbox. Observe that the quotes around the "John D. Smith" example were required, on account of the '.' character, and they would also have been required had any UTF8-xtra-char been present. 5.3. Message-ID The Message-ID header contains the article's message identifier, a unique identifier distinguishing the article from every other article. The content syntax makes use of syntax defined in [RFC 2822], subject to the following revised definition of no-fold-quote and no-fold-literal. Message-ID-content = msg-id id-left = dot-atom-text / no-fold-quote id-right = dot-atom-text / no-fold-literal no-fold-quote = DQUOTE *( strict-qtext / strict-quoted-pair ) DQUOTE no-fold-literal = DQUOTE *( dtext / strict-quoted-pair ) DQUOTE A msg-id MUST NOT contain any SP within any strict-quoted-pair. The msg-id MUST NOT be more than 250 octets in length. NOTE: The syntax ensures that a msg-id is restricted to pure US-ASCII, and is thus a strict subset of that defined by [RFC 2822]. The exclusion of SP is to ensure compatibility with existing software. The length restriction ensures that systems which accept message identifiers as a parameter when retrieving an article (e.g. [NNTP]) can rely on a bounded length. Observe that msg-id includes the '<' and '>'. Following the provisions of [RFC 2822], an agent generating an article's message identifier MUST ensure that it is unique and that it is NEVER reused (either in Netnews or email). Moreover, even though commonly derived from the domain name of the originating site (and domain names are case-insensitive), a message identifier MUST NOT be altered in any way during transport, or when copied (as into a References header), and thus a simple (case-sensitive) comparison of octets will always suffice to recognise that same message identifier wherever it subsequently reappears. NOTE: some old software may treat message identifiers that differ only in case within their id-right part as equivalent, and implementors of agents that generate message identifiers should be aware of this. 5.4. Subject The Subject header contains a short string identifying the topic of the message. This is an inheritable header (4.2.2.2) to be copied into the Subject header of any followup, in which case the new header-content SHOULD then default to the string "Re: " (a "back C. H. Lindsey [Page 29] News Article Format July 2001 reference") followed by the contents of the pure-subject of the precursor. Any leading "Re: " in the pure-subject MUST be stripped. Subject-content = [ back-reference ] pure-subject pure-subject = 1*( [FWS] utext ) back-reference = %x52.65.3A.20 ; which is a case-sensitive "Re: " The pure-subject MUST NOT begin with "Re: ". NOTE: The given syntax differs from that prescribed in [RFC 2822] insofar as it does not permit a header content to be completely empty, or to consist of WSP only (see remarks in 4.2.5 concerning undesirable headers). Followup agents MAY remove strings that are known to be used erroneously as back-reference (such as "Re(2): ", "Re:", "RE: ", or "Sv: ") from the Subject-content when composing the subject of a followup and add a correct back-reference in front of the result. NOTE: that would be "SHOULD remove instances" except that we cannot find a sufficiently robust and simple algorithm to do the necessary natural language processing. Followup agents MUST NOT use any other string except "Re: " as a back reference. Specifically, a translation of "Re: " into a local language or usage MUST NOT be used. NOTE: "Re" is an abbreviation for the Latin "In re", meaning "in the matter of", and not an abbreviation of "Reference" as is sometimes erroneously supposed. Agents SHOULD NOT depend on nor enforce the use of back references by followup agents. For compatibility with legacy news software the Subject-content of a control message (i.e. an article that also contains a Control header) MAY start with the string "cmsg ", and non-control messages MUST NOT start with the string "cmsg ". See also section 6.13. 5.4.1. Examples In the following examples, please note that only "Re: " is mandated by this standard. "was: " is a convention used by many English- speaking posters to signal a change in subject matter. Software should be able to deduce this information from References. Subject: Film at 11 Subject: Re: Film at 11 Subject: Godwin's law considered harmful (was: Film at 11) Subject: Godwin's law (was: Film at 11) Subject: Re: Godwin's law (was: Film at 11) C. H. Lindsey [Page 30] News Article Format July 2001 5.5. Newsgroups The Newsgroups header's content specifies the newsgroup(s) in which the article is intended to appear. It is an inheritable header (4.2.2.2) which then becomes the default Newsgroups header of any followup, unless a Followup-To header is present to prescribe otherwise. References to "Unicode" or "the latest version of the Unicode Standard" mean [UNICODE 3.1] contains guarantees of strict future upwards compatibility (e.g. no character will be removed or change classification). Implementors should be aware that currently unassigned code points (Unicode category Cn) may become valid characters in future versions of Unicode. Since the poster of an article might have access to a newer version of that standard, relaying and serving agents MUST accept such characters, but posting agents (and indeed all agents) MUST NOT generate them. Newsgroups-content = newsgroup-name *( *FWS ng-delim *FWS newsgroup-name ) *FWS newsgroup-name = component *( "." component ) component = 1*component-glyph ng-delim = "," component-glyph = combiner-base *combiner-mark combiner-base = combiner-ASCII / combiner-extended combiner-ASCII = "0"-"9" / %x41-5A / %x61-7A / "+" / "-" / "_" combiner-extended = combiner-mark = NOTE: the excluded characters are control characters (Cc), format control characters (Cf), surrogates (Cs), and separators (Zs, Zl, Zp). In particular, this excludes all whitespace characters. Each component MUST be invariant under Unicode normalization NFKC (cf. the weaker normalization requirement for other headers in section 4.4.1 which specified no more than normalization NFC). NOTE: Alternatively, this restriction could have been expressed by saying: o All characters with a compatibility decomposition are forbidden; or else o All characters with property NFKC-NO are forbidden. The effect is to exclude variant forms of characters, such as superscripts and subscripts, wide and narrow forms, font variants, encircled forms, ligatures, and so on, as their use could cause confusion. C. H. Lindsey [Page 31] News Article Format July 2001 As a result of of this restriction, a name has only one valid form. Implementations can assume that a straight comparison of characters or octets is sufficient to compare two newsgroup- names. NOTE: An implementation is not required to apply NFKC, or any other normalization, to newsgroup names. Only agencies that create new groups need to be careful to obey this restriction (7.1). However, if a posting agent neglects to normalize a newsgroup-name entered manually, this may lead to the user posting to a non-existent group without understanding why. Newsgroup-names containing non-ASCII characters MUST be encoded in UTF-8 and not according to [RFC 2047]. Components beginning with underline ("_") are reserved for use by future versions of this standard and MUST NOT occur in newsgroup names (whether in Newsgroup headers or in newgroup control messages (7.1)). However, such names MUST be accepted. Components beginning with "+" or "-" are reserved for use by implementations and MUST NOT occur in newsgroup names (whether in Newsgroup headers or in newgroup control messages). Implementors may assume that this rule will not change in any future version of this standard. NOTE: For example, implementors may safely use leading "+" and "-" to "escape" other entities within something that looks like a newsgroup-name. Agencies responsible for the administration of particular hierarchies Ought to place additional restrictions on the characters they allow in newsgroup-names within those hierarchies (such as to accord with the languages commonly used within those hierarchies, or to avoid perceived ambiguities pertinent to those languages). Where there is no such specific policy, the following restrictions SHOULD be applied to newsgroup names. NOTE: These restrictions are intended to reflect existing practice, with some additions to accomodate foreseeable enhancements, and are intended both to avoid certain technical difficulties and to avoid unnecessary confusion. It may well be that experience will allow future extensions to this standard to relax some or all of these restrictions. The specific restrictions (to be applied in the absence of established policies to the contrary) are: 1. The following characters are forbidden, subject to the comments and notes at the end of the list: characters in category Cn (Other, Not assigned) [1] characters in category Co (Other, Private Use) [2] characters in category Lt (Letter, Titlecase) [3] C. H. Lindsey [Page 32] News Article Format July 2001 characters in category Lu (Letter, Uppercase) [3] characters in category Me (Mark, Enclosing) [4] characters in category Pd (Punctuation, Dash) [4][5] characters in category Pe (Punctuation, Close) [4] characters in category Pf (Punctuation, Final quote) [4] characters in category Pi (Punctuation, Initial quote) [4] characters in category Po (Punctuation, Other) [4] characters in category Ps (Punctuation, Open) [4] characters in category Sc (Symbol, Currency) [4] characters in category Sk (Symbol, Modifier) [4] characters in category Sm (Symbol, Math) [4][5] characters in category So (Symbol, Other) [4] [1] As new characters are added to Unicode, the code point moves from category Cn to some other category. As stated above, implementors should be prepared for this. [2] Specific private use characters can be used within a hierarchy or co-operating subnet that has agreed meanings for them. [3] Traditionally, newsgroup-names have been written in lowercase. Posting agents MAY convert these characters to the corresponding lowercase forms. [That may be better left unsaid, or rewritten] [4] Traditionally newsgroup names have only used letters, digits, and the three special characters "+", "-" and "_". These categories correspond to characters outside that set. [5] Although the characters "+" and "-" are within categories Pd and Sm, they are not forbidden. 2. A component name is forbidden to consist entirely of digits. NOTE: This requirement was in [RFC 1036] but nevertheless several such groups have appeared in practice and implementors should be prepared for them. A common implementation technique uses each component as the name of a directory and uses numeric filenames for each article within a group. Such an implementation needs to be careful when this could cause a clash (e.g. between article 123 of group xxx.yyy and the directory for group xxx.yyy.123). [Open issue a number of people think this should not be a default requirement but simply be a NOTE; wording for such is further down.] 3. A component is limited to 30 component-glyphs and a newsgroup-name to 71 component-glyphs. Whilst there is no longer any technical reason to limit the length of a component (formerly, it was limited to 14 octets) nor of a newsgroup-name, it should be noted that these names are also used in the newsgroups line (7.1.2) where an overall policy limit applies and, moreover, excessively long names can be exceedingly inconvenient in practical use. C. H. Lindsey [Page 33] News Article Format July 2001 NOTE: To all intents and purposes, a component-glyph is what a user might regard as a single "character" as displayed on his screen, though it might be transmitted as several actual characters (e.g. q-circumflex is two characters). Serving and relaying agents MUST accept any newsgroup-name that meets the above requirements, even if they violate one or more of the policy restrictions. Posting and injecting agents MAY reject articles containing newsgroup-names that do not meet these restrictions, and posting agents MAY attempt to correct them (e.g. by lowercasing). However, because of the large and changing tables required to do these checks and corrections throughout the whole of Unicode, this standard does not require them to do so. Rather, the onus is placed on those who create new newsgroups (7.1) to check the mandatory requirements, to consider the effects of relaxing the other restrictions, and to consider how all this may affect propagation of the group. Since future extensions to this standard and the Unicode standard, plus any relaxations of the default restrictions introduced by specific hierarchies, might invalidate some such checks, warnings, and adjustments, implementations MUST incorporate means to disable them. In particular, implementations must be prepared for a relaxation of the normalization requirements (e.g. from NFKC down to NFC), which have been made rather stringent due to a lack of practical experience in this area. [Alternative text for Open issue] NOTE: Components composed entirely of digits were forbidden by [RFC 1036] but have nevertheless been used in practice, and are therefore permitted by this specification. A common implementation technique uses each component as the name of a directory and uses numeric filenames for each article within a group. Such an implementation needs to be careful when this could cause a clash (e.g. between article 123 of group xxx.yyy and the directory for group xxx.yyy.123). [Open issue: delete the above text if we retain the default requirement above.] NOTE: The newsgroup-name as encoded in UTF-8 should be regarded as the canonical form. Reading agents may convert it to whatever character set they are able to display (see 4.4.1) and serving agents may possibly need to convert it to some form more suitable as a filename. Simple algorithms for both kinds of conversion are readily available. Observe that the syntax does not allow comments within the Newsgroups header; this is to simplify processing by relaying and serving agents which have a requirement to process this header extremely rapidly. The inclusion of folding white space within a Newsgroups-content is a newly introduced feature in this standard. It MUST be accepted by all conforming implementations (relaying agents, serving agents and reading agents). Posting agents should be aware that such postings C. H. Lindsey [Page 34] News Article Format July 2001 may be rejected by overly-critical old-style relaying agents. When a sufficient number of relaying agents are in conformance, posting agents SHOULD generate such whitespace in the form of so as to keep the length of lines in the relevant headers (notably Newsgroups and Followup-To) to no more than than 79 characters (or other agreed policy limit - see 4.5). Before such critical mass occurs, injecting agents MAY reformat such headers by removing whitespace inserted by the posting agent, but relaying agents MUST NOT do so. Posters SHOULD use only the names of existing newsgroups in the Newsgroups header. However, it is legitimate to cross-post to newsgroup(s) which do not exist on the posting agent's host, provided that at least one of the newsgroups DOES exist there, and followup agents SHOULD accept this (posting agents MAY accept it, but Ought at least to alert the poster to the situation and request confirmation). Relaying agents MUST NOT rewrite Newsgroups headers in any way, even if some or all of the newsgroups do not exist on the relaying agent's host. Serving agents MUST NOT create new newsgroups simply because an unrecognised newsgroup-name occurs in a Newsgroups header (see 7.1 for the correct method of newsgroup creation). The Newsgroups header is intended for use in Netnews articles rather than in mail messages. It MAY be used in a mail message to indicate that it is a copy also posted to the listed newsgroups, but it SHOULD NOT be used in a mail-only reply to a Netnews article (thus the "inheritable" property of this header applies only to followups to a newsgroup, and not to followups to the poster). Moreover, if a newsgroup-name contains any non-ASCII character, it MAY be encoded using the mechanism defined in [RFC 2047] when sent by mail but, if it is subsequently returned to the Netnews environment, it MUST then be re-encoded into UTF-8. 5.5.1. Forbidden newsgroup names The following forms of newsgroup-name MUST NOT be used except for the specific purposes indicated: o Newsgroup-names having only one component. These are reserved for newsgroups whose propagation is restricted to a single host or local network, and for pseudo-newsgroups such as "poster" (which has special meaning in the Followup-To header - see section 6.7), "junk" (often used by serving agents), "control" (likewise), "revise" and "repost" (which have special meanings in the Xref header - see 6.16) o Any newsgroup-name beginning with "control." (used as pseudo- newsgroups by many serving agents) o Any newsgroup-name containing the component "ctl" (likewise) o "to" or any newsgroup-name beginning with "to." (reserved for the ihave/sendme protocol described in section 7.6, and for test messages sent on an essentially point-to-point basis) C. H. Lindsey [Page 35] News Article Format July 2001 o Any newsgroup-name beginning with "example." (reserved for examples in this and other standards) o Any newsgroup-name containing the component "all" (because this is used as a wildcard in some implementations) A newsgroup-name SHOULD NOT appear more than once in the Newsgroups header. The order of newsgroup names in the Newsgroups header is not significant, except for determining which moderator to send the article to if more than one of the groups is moderated (see 8.2). 5.6. Path The Path header shows the route taken by a message since its entry into the Netnews system. It is a variant header (4.2.2.3), each agent that processes an article being required to add one (or more) entries to it. This is primarily to enable relaying agents to avoid sending articles to sites already known to have them, in particular the site they came from, and additionally to permit tracing the route articles take in moving over the network, and for gathering Usenet statistics. Finally the presence of a '%' delimiter in the Path header can be used to identify an article injected in conformance with this standard. 5.6.1. Format Path-content = *( path-identity [FWS] delimiter [FWS] ) tail-entry *FWS path-identity = 1*( ALPHA / DIGIT / "-" / "." / ":" / "_" ) delimiter = "/" / "?" / "%" / "," / "!" tail-entry = 1*( ALPHA / DIGIT / "-" / "." / ":" / "_" ) NOTE: A Path-content will inevitably contain at least one path- identity, except possibly in the case of a proto-article that has not yet been injected onto the network. NOTE: Observe that the syntax does not allow comments within the Path header; this is to simplify processing by relaying and injecting agents which have a requirement to process this header extremely rapidly. A relaying agent SHOULD NOT pass an article to another relaying agent whose path-identity (or some known alias thereof) already appears in the Path-content. Since the comparison may be either case sensitive or case insensitive, relaying agents SHOULD NOT generate a name which differs from that of another site only in terms of case. A relaying agent MAY decline to accept an article if its own path- identity is already present in the Path-content or if the Path- content contains some path-identity whose articles the relaying agent does not want, as a matter of local policy. NOTE: This last facility is sometimes used to detect and decline control messages (notably cancel messages) which have been deliberately seeded with a path-identity to be "aliased out" by C. H. Lindsey [Page 36] News Article Format July 2001 sites not wishing to act upon them. 5.6.2. Adding a path-identity to the Path header When an injecting, relaying or serving agent receives an article, it MUST prepend its own path-identity followed by a delimiter to the beginning of the Path-content. In addition, it SHOULD then add CRLF and WSP if it would otherwise result in a line longer than 79 characters. The path-identity added MUST be unique to that agent. To this end it SHOULD be one of: 1. A fully qualified domain name (FQDN) associated (by the Internet DNS service [RFC 1034]) with an A record, which SHOULD identify the actual machine prepending this path-identity. Ideally, this FQDN should also be "mailable" in the sense that it enables the construction of a valid E-mail address of the form "usenet@" or "news@" [RFC 2142] whereby the administrators of that agent may be reached. 2. A fully qualified domain name (FQDN) associated (by the Internet DNS service) with an MX record which MUST then enable the construction of a valid E-mail address of the form "usenet@" or "news@" whereby the administrators of that agent may be reached. 3. An arbitrary name believed to be unique and registered at least with all sites immediately downstream from the given site. 4. An encoding of an IP address - [RFC 820] or [RFC 2373] (the requirement to be able to use an is the reason for including ':' as an allowed character within a path-identity). Of the above options, nos. 1 to 3 are much to be preferred, unless there are strong technical reasons dictating otherwise. In particular, the injecting agent's path-identity MUST, as a special case, be an FQDN mailable address in the sense defined under option 1, or with an associated MX record as in option 2. The injecting agent's path-identity MUST be followed by the special delimiter '%' which serves to separate the pre-injection and post- injection regions of the Path-content (see 5.6.3). In the case of a relaying or serving agent, the delimiter is chosen as follows. When such an agent receives an article, it MUST establish the identity of the source and compare it with the leftmost path-identity of the Path-content. If it matches, a '/' should be used as the delimiter when prepending the agent's own path-identity. If it does not match then the agent should prepend two entries to the Path-content; firstly the true established path-identity of the source followed by a '?' delimiter, and then, to the left of that, the agent's own path-identity followed by a '/' delimiter as usual. C. H. Lindsey [Page 37] News Article Format July 2001 This prepending of two entries SHOULD NOT be done if the provided and established identities match. Any method of establishing the identity of the source may be used (but see 5.6.5 below), with the consideration that, in the event of problems, the agent concerned may be called upon to justify it. NOTE: The use of the '%' delimiter marks the position of the injecting agent in the chain. In normal circumstances there should therefore be only one `%` delimiter present, and injecting agents MAY choose to reject proto-articles with a '%' already in them. If, for whatever reason, more than one '%' is found, then the path-identity in front of the leftmost '%' is to be regarded as the true injecting agent. 5.6.3. The tail-entry For historical reasons, the tail-entry (i.e. the rightmost entry in the Path-content) is regarded as a "user name", and therefore MUST NOT be interpreted as a site through which the article has already passed. Moreover, the Path-content is not an E-mail address and MUST NOT be used to contact the poster. Posting and/or injecting agents MAY place any string here. When it is not an actual user name, the string "not-for-mail" is often used, but in fact a simple "x" would be sufficient. Often this field will be the only entry in the region (known as the pre-injection region) after the '%', although there may be entries corresponding to machines traversed between the posting agent and the injecting agent proper. In particular, injecting agents that receive articles from many sources MAY include information to establish the circumstances of the injection such as the identity of the source machine (especially if the Injector-Info header (6.19) is absent). Any such inclusion SHOULD NOT conflict with any genuine site identifier. The '!' delimiter may be used freely within the pre- injection region, although '/' and '?' are also appropriate if used correctly. 5.6.4. Delimiter Summary A summary of the various delimiters. The name immediately to the left of the delimiter is always that of the machine which added the delimiter. '/' The name immediately to the right is known to be the identity of the machine from which the article was received (either because the entry was made by that machine and we have verified it, or because we have added it ourselves). '?' The name immediately to the right is the claimed identity of the machine from which the article was received, but we were unable to verify it (and have prepended our own view of where it came from, and then a '/'). C. H. Lindsey [Page 38] News Article Format July 2001 '%' Everything to the right is the pre-injection region followed by the tail-entry. The name on the left is the FQDN of the injecting agent. The presence of two '%'s in a path indicates a double-injection (see 8.2.2). '!' The name immediately to the right is unverified. The presence of a '!' to the left of the '%' indicates that the identity to the left is that of an old-style system not conformant with this standard. ',' Reserved for future use, treat as '/'. Other Old software may possibly use other delimiters, which should be treated as '!'. But note in particular that ':', '-' and '_' are components of names, not delimiters, and FWS on its own MUST NOT be used as the sole delimiter. NOTE: Old Netnews relaying and injecting programs almost all delimit Path entries with the '!' delimiter, and these entries are not verified. As such, the presence of '%' as a delimiter will indicate that the article was injected by software conforming to this standard, and the presence of '!' as a delimiter to the left of a '%' will indicate that the message passed through systems developed prior to this standard. It is anticipated that relaying agents will reject articles in the old style once this new standard has been widely adopted. 5.6.5. Suggested Verification Methods The following approaches for common transports are suggested in order to meet a site's verification obligations. They are not required, but following them should avoid the necessity for wasteful double-entry Path additions. If the incoming article arrives through some TCP/IP protocol such as NNTP, the IP address of the source will be known, and will likely already have been checked against a list of known FQDNs or IP addresses that the receiving site has agreed to peer with (this will have involved a DNS lookup of a known FQDN, following CNAME chains as required, to find an A record containing that source IP). 1. Where the path-identity is an FQDN (or even an arbitrary name starting with a '.') it is now a simple matter to check that it is the proper FQDN for the source, or some known registered alias thereof. Alternatively, where the FQDN in the path-identity has an associated A record, an immediate DNS lookup as above can be used to verify it. 2. Where the path-identity is an encoding of an IP address which does not immediately match the known IP address of the source, a reverse-DNS (in-addr.arpa PTR record) lookup may be done on the provided address, followed by a regular DNS "A" record lookup on the returned name. There may be A records for several IP C. H. Lindsey [Page 39] News Article Format July 2001 addresses, of which one should match the path-identity and another should match the source. 3. If the path-identity fails to match any known alias for the source (requiring the insertion of an extra path-identity for the true source followed by a '?'), simply doing a reverse DNS (PTR) lookup on the source IP address is not sufficient to generate the true FQDN. The returned name must be mapped back to A records to assure it matches the source's IP address. If the incoming article arrives through some other protocol, such as UUCP, that protocol MUST include a means of verifying the source site. In UUCP implementations, commonly each incoming connection has a unique login name and password, and that login name (or some alias registered for it) would be expected as the path-identity. [The above description may still contain more detail that we would wish. My aim so far was to retain everything in Brad's original, but expressed in a more palatable manner. We can now decide how much of it we want to keep.] 5.6.6. Example Path: foo.isp.example/ foo-server/bar.isp.example?10.123.12.2/old.site.example! barbaz/baz.isp.example%dialup123.baz.isp.example!x NOTE: That article was injected into the news stream by baz.isp.example (complaints may be addressed to usenet@baz.isp.example). The injector has taken care to record that it got it from dialup123.baz.isp.example. "x" is the default tail entry, though sometimes a real userid is put there. The article was relayed, perhaps by UUCP, to the machine known, at least to its downstream, as "barbaz". Barbaz relayed it to old.site.example, which does not yet conform to this standard (hence the '!' delimiter). So one cannot be sure that it really came from barbaz. Old.site.example relayed it to a site claiming to have the IP address [10.123.12.2], and claiming (by using the '/' delimiter) to have verified that it came from old.site.example. [10.123.12.2] relayed it to "foo-server" which, not being convinced that it truly came from [10.123.12.2], did a reverse lookup on the actual source and concluded it was known as bar.isp.example (that is not to say that [10.123.12.2] was not a correct IP address for bar.isp.example, but simply that that connection could not be substantiated by foo-server). Observe that foo-server has now added two entries to the Path. C. H. Lindsey [Page 40] News Article Format July 2001 "foo-server" is a locally significant name within the complex site of many machines run by foo.isp.example, so the latter should have no problem recognizing foo-server and using a '/' delimiter. Presumably foo.isp.example then delivered the article to its direct clients. It appears that foo.isp.example and old.site.example decided to fold the line, on the grounds that it seemed to be getting a little too long. 6. Optional Headers The headers appearing in this section have established meanings and MUST be interpreted according to the definitions given here. None of them is required to appear in every article but some of them are required in certain types of article, such as followups. Any header defined in this (or any other) standard MUST NOT appear more than once in an article unless specifically stated otherwise. Experimental headers (4.2.2.1) and headers defined by cooperating subnets are exempt from this requirement. See section 8 "Duties of Various Agents" for the full picture. 6.1. Reply-To The Reply-To header specifies a reply address(es) to be used for personal replies for the poster(s) of the article when this is different from the poster's address(es) given in the From header. The content syntax makes use of syntax defined in [RFC 2822], but subject to the revised definition of local-part given in section 5.2. Reply-To-content = From-content ; see 5.2 In the absence of Reply-To, the reply address(es) is the address(es) in the From header. For this reason a Reply-To SHOULD NOT be included if it just duplicates the From header. NOTE: Use of a Reply-To header is preferable to including a similar request in the article body, because reply agents can take account of Reply-To automatically. An address of "<>" in the Reply-To header MAY be used to indicate that the poster does not wish to recieve email replies. 6.1.1. Examples Reply-To: John Smith Reply-To: John Smith , dave@isp.example Reply-To: John Smith ,andrew@isp.example, fred@site2.example Reply-To: Please do not reply <> C. H. Lindsey [Page 41] News Article Format July 2001 6.2. Sender The Sender header specifies the mailbox of the entity which actually sent this article, if that entity is different from that given in the From header or if more than one address appears in the From header. This header SHOULD NOT appear in an article unless the sender is different from the poster. This header is appropriate for use by automatic article posters. The content syntax makes use of syntax defined in [RFC 2822]. Sender-content = mailbox 6.3. Organization The Organization header is a short phrase identifying the poster's organization. Organization-content= 1*( [FWS] utext ) NOTE: Posting and injecting agents are discouraged from providing a default value for this header unless it is acceptable to all posters using those agents. Unless this header contains useful information (including some indication of the posters physical location) posters are discouraged from including it. 6.4. Keywords The Keywords field contains a comma separated list of important words and phrases intended to describe some aspect of the content of the article. The content syntax makes use of syntax defined in [RFC 2822]. Keywords-content = phrase *( "," phrase ) NOTE: The list is comma separated NOT space separated. 6.5. Summary The Summary header is a short phrase summarizing the article's content. Summary-content = 1*( [FWS] utext ) The summary should be terse. Authors Ought to avoid trying to cram their entire article into the headers; even the simplest query usually benefits from a sentence or two of elaboration and context, and not all reading agents display all headers. On the other hand the summary should give more detail than the Subject. C. H. Lindsey [Page 42] News Article Format July 2001 6.6. Distribution The Distribution header is an inheritable header (see 4.2.2.2) which specifies geographical or organizational limits to an article's propagation. Distribution-content= distribution *( dist-delim distribution ) dist-delim = "," distribution = positive-distribution / negative-distribution positive-distribution = *FWS distribution-name *FWS negative-distribution = *FWS "!" distribution-name *FWS distribution-name = ALPHA 1*distribution-rest distribution-rest = ALPHA / "+" / "-" / "_" NOTE: The use of ALPHA in the syntax ensures that distribution names are always in US-ASCII. Articles MUST NOT be passed between relaying agents or to serving agents unless the sending agent has been configured to supply and the receiving agent to receive BOTH of (a) at least one of the newsgroups in the article's Newsgroups header, and (b) at least one of the positive-distributions (if any) in the article's Distribution header and none of the negative- distributions. Additionally, reading agents MAY be configured so that unwanted distributions do not get displayed. NOTE: Although it would seem redundant to filter out unwanted distributions at both ends of a relaying link (and it is clearly more efficient to do so at the sending end), many sending sites have been reluctant, historically speaking, to apply such filters (except to ensure that distributions local to their own site or cooperating subnet did not escape); moreover they tended to configure their filters on an "all but those listed" basis, so that new and hitherto unheard of distributions would not be caught. Indeed many "hub" sites actually wanted to receive all possible distributions so that they could feed on to their clients in all possible geographical (or organizational) regions. Therefore, it is desirable to provide facilities for rejecting unwanted distributions at the receiving end. Indeed, it may be simpler to do so locally than to inform each sending site of what is required, especially in the case of specialized distributions (for example for control messages, such as cancels from certain issuers) which might need to be added at short notice. Tha possibility for reading agents to filter distributions has been provided for the same reason. C. H. Lindsey [Page 43] News Article Format July 2001 Exceptionally, ALL relaying agents are deemed willing to supply or accept the distribution "world", and NO relaying agent should supply or accept the distribution "local". However, "world" SHOULD NEVER be mentioned explicitly since it is the default when the Distribution header is absent entirely. "All" MUST NOT be used as a distribution-name. Distribution-names SHOULD contain at least three characters, except when they are two-letter country names as in [ISO 3166]. Distribution-names are case-insensitive (i.e. "US", "Us" and "us" all specify the same distribution). NOTE: "Distribution: !us" can be used to cause an article to go to the whole of "world" except for "us". Posting agents Ought Not to provide a default Distribution header without giving the poster an opportunity to override it. Followup agents SHOULD initially supply the same Distribution header as found in the precursor. 6.7. Followup-To The Followup-To header specifies which newsgroup(s) followups should be posted to. Followup-To-content = Newsgroups-content / "poster" The syntax is the same as that of the Newsgroups-content, with the exception that the magic word "poster" is allowed. In the absence of a Followup-To header, the default newsgroup(s) for a followup are those in the Newsgroups header, and for this reason the Followup-To header SHOULD NOT be included if it just duplicates the Newsgroups header. A Followup-To header consisting of the magic word "poster" indicates that the poster requests no followups to be sent in response to this article, only personal replies to the article's reply address. NOTE: A poster who wishes both a personal reply and a followup post should include a Mail-Copies-To header (6.8). 6.8. Mail-Copies-To The Mail-Copies-To header indicates whether or not the poster wishes to have followups to an article emailed in addition to being posted to Netnews and, if so, establishes the address to which they should be sent. The content syntax makes use of syntax defined in [RFC 2822], but subject to the revised definition of local-part given in section 5.2. Mail-Copies-To-content = copy-addr / "nobody" / "poster" copy-addr = mailbox C. H. Lindsey [Page 44] News Article Format July 2001 The keyword "nobody" indicates that the poster does not wish copies of any followup postings to be emailed. This indication is widely seen as a very strong wish, and is to be taken as the default when this header is absent. The keyword "poster" indicates that the poster wishes a copy of any followup postings to be emailed to him. Otherwise, this header contains a copy-addr to which the poster wishes a copy of any followup postings to be sent. NOTE: Some existing practice uses the keyword "never" in place of "nobody" and "always" in place of "poster". These usages are deprecated, but followup agents MAY observe them. The automatic actions of a followup agent in the various cases (subject to manual override by the user) are as follows: nobody (or when the header is absent) The followup agent SHOULD NOT, by default, email such a copy and Ought, especially when there is an explicit "nobody", to issue a warning and ask for confirmation if the user attempts to do so. poster The followup agent Ought, by default, to email a copy, which MUST then be sent to the address in the Reply-To header, and in the absence of that to the address(es) in the From header. copy-addr The followup agent Ought, by default, to email a copy, which MUST then be sent to the copy-addr. NOTE: This header is only relevant when posting followups to Netnews articles, and is to be ignored when sending pure email replies to the poster, which are handled as prescribed under the Reply-To header (6.1). Whether or not this header will also find similar usage for replies to messages sent to mailing lists falls outside the scope of this standard. When emailing a copy, the followup agent SHOULD also include a "Posted-And-Mailed: yes" header (6.9). NOTE: In addition to the Posted-And-Mailed header, some followup agents also include within the body a mention that the article is both posted and mailed, for the benefit of reading agents that do not normally show that header. 6.9. Posted-And-Mailed Posted-And-Mailed-content = "yes" / "no" This header, when used with the "yes" keyword, indicates that the article has been both posted to the specified newsgroups and emailed. It SHOULD be used when replying to the poster of an article to which C. H. Lindsey [Page 45] News Article Format July 2001 this one is a followup (see the Mail-Copies-To header in section 6.8) and it MAY be used when any article is also mailed to a re