THE CODEBREAKERS
SCRIBNER
1230 Avenue of the Americas
New York, NY 10020
www.SimonandSchuster.com
Copyright © 1967, 1996 by David Kahn
All rights reserved, including the right of reproduction in whole or in part in any form.
SCRIBNER and design are trademarks of Macmillan Library Reference USA, Inc. under license by Simon & Schuster, the publisher of this work.
Manufactured in the United States of America
13 15 17 19 20 18 16 14 12
Library of Congress Cataloging-in-Publication Data is available.
ISBN 0-684-83130-9
ISBN: 978-0-684-83130-5
eISBN: 978-1-439-10355-5
Dedication
To my Parents and my Grandmother
CONTENTS
Preface to the Revised Edition
Preface
A Few Words
1. One Day of Magic
THE PAGENT OF CRYPTOLOGY
2. The First 3,000 Years
3. The Rise of the West
4. On the Origin of a Species
5. The Era of the Black Chambers
6. The Contribution of the Dilettantes
7. Crises of the Union
8. The Professor, the Soldier, and the Man on Devil’s Island
9. Room 40
10. A War of Intercepts: I
11. A War of Intercepts: II
12. Two Americans
13. Secrecy for Sale
14. Duel in the Ether: The Axis
15. Duel in the Ether: Neutrals and Allies
16. Censors, Scramblers, and Spies
17. The Scrutable Orientals
18.
19. N.S.A.
SIDESHOWS
20. The Anatomy of Cryptology
21. Heterogeneous Impulses
22. Rumrunners, Businessmen, and Makers of Non-secret Codes
23. Ciphers in the Past Tense
24. The Pathology of Cryptology
PARACRYPTOLOGY
25. Ancestral Voices
26. Messages from Outer Space
THE NEW CRYPTOLOGY
27. Cryptology Goes Public
Bibliography
Notes to Text
Acknowledgments
Notes to Illustrations
Index
PREFACE TO THE REVISED EDITION
The need to revise this book existed even before it was published on September 27, 1967. I had written what I hoped would be the definitive history of the subject. I did not know at the time of such great matters as the Polish-British-American mastery of the German Enigma cipher machine, which had such great effects on World War II, or of such lesser ones as the tactical value of German front-line telephone taps. Nor did I—or anyone—know of things that had not yet been invented, such as public-key cryptography. The first glimmering that the world of cryptology would not stand still for me came four months after publication, when North Korea seized the U.S. electronic reconnaissance ship Pueblo in January 1968. It marked the first of a series of events that showed the need for revision. I had, indeed, made some minor corrections in printings three through seven, but then I concentrated on other projects.
There followed, however, the Ultra disclosures, the creation of public-key cryptography, and the enormous growth in computer communications, including particularly the appearance of the Internet, where cryptography affords the best means for privacy. At about the same time, the absorption of Macmillan, the original publisher, by Simon & Schuster brought a young, energetic editor named Scott Moyers to handle The Codebreakers. He saw that I could fulfill my obligation to cryptology and at the same time help the book sell better by incorporating the new material as a single chapter. This made sense, and that is what I’ve done.
I have sought to cover the major events, both external and internal, that have affected cryptology in the past quarter century. It is amazing how much these have changed the field. Fortunately for me, while they have added information, they do not change the past, so the first edition remains valid. I hope that this new edition will prove as useful—and perhaps as pleasurable—to readers as the previous one.
DAVID KAHN
Great Neck, New York
May 1996
PREFACE
CODEBREAKING is the most important form of secret intelligence in the world today. It produces much more and much more trustworthy information than spies, and this intelligence exerts great influence upon the policies of governments. Yet it has never had a chronicler.
It badly needs one. It has been estimated that cryptanalysis saved a year of war in the Pacific, yet the histories give it but passing mention. Churchill’s great history of World War II has been cleaned of every single reference to Allied communications intelligence except one (and that based on the American Pearl Harbor investigation), although Britain thought it vital enough to assign 30,000 people to the work. The intelligence history of World War II has never been written. All this gives a distorted view of why things happened. Furthermore, cryptology itself can benefit, like other spheres of human endeavor, from knowing its major trends, its great men, its errors made and lessons learned.
I have tried in this book to write a serious history of cryptology. It is primarily a report to the public on the important role that cryptology has played, but it may also orient cryptology with regard to its past and alert historians to the sub rosa influence of cryptanalysis. The book seeks to cover the entire history of cryptology. My goal has been twofold: to narrate the development of the various methods of making and breaking codes and ciphers, and to tell how these methods have affected men.
When I began this book, I, like other well-informed amateurs, knew about all that had been published on the history of cryptology in books on the subject. How little we really knew! Neither we nor any professionals realized that many valuable articles lurked in scholarly journals, or had induced any cryptanalysts to tell their stories for publication, or had tapped the vast treasuries of documentary material, or had tried to take a long view and ask some questions that now appear basic. I believe it to be true that, from the point of view of the material previously published in books on cryptology, what is new in this book is 85 to 90 per cent.
Yet it is not exhaustive. A foolish secrecy still clothes much of World War II cryptology—though I believe the outlines of the achievements are known—and to tell just that story in full would require a book the size of this. Even in, say, the 18th century, the unexplored manuscript material is very great.
Nor is this a textbook. I have explained at length only two basic methods of solution, though I have sketched many others. For some readers even this will be too much; them I advise to skip this material. They will not have a full understanding of what is going on, but that will not cripple their comprehension of the stories. For readers who want more detail on these methods, I recommend Helen F. Gaines’s Elementary Cryptanalysis, partly because it is a competent work, partly because it is the only work of its kind in English now easily available (in a paperback reprint, entitled Cryptanalysis). In French, there is Luigi Sacco’s outstanding Manuel de cryptographie (the Italian original is out of print). Nearly all the other books in print are juveniles. Readers interested in cryptanalysis may also join the American Cryptogram Association, which publishes a magazine with articles on how to solve ciphers and with cryptograms for solution.
In my writing, I have tried to adhere to two principles. One was to use primary sources as much as possible. Often it could not be done any other way, since nothing had been published on a particular matter. The other principle was to try to make certain that I did not give cryptology sole and total credit for winning a battle or making possible a diplomatic coup or whatever happened if,
as was usual, other factors played a role. Narratives which make it appear as if every event in history turned upon the subject under discussion are not history but journalism. They are especially prevalent in spy stories, and cryptology is not immune. The only other book-length attempt to survey the history of cryptology, the late Fletcher Pratt’s Secret and Urgent, published in 1939, suffers from a severe case of this special pleading. Pratt writes thrillingly—perhaps for that very reason—but his failure to consider the other factors, together with his errors and omissions, his false generalizations based on no evidence, and his unfortunate predilection for inventing facts vitiate his work as any kind of a history. (Finding this out was disillusioning, for it was this book, borrowed from the Great Neck Library, that interested me in cryptology.) I think that although trying to balance the story with the other factors may detract a little from the immediate thrill, it charges it with authenticity and hence makes for long-lasting interest: for this is how things really happened.
In the same vein, I have not made up any conversations, and my speculations about things not a matter of record have been marked as such in the notes. I have documented all important facts, except that in a few cases I have had to respect the wishes of my sources for anonymity.
The manuscript was submitted to the Department of Defense on March 4, 1966.
It is impossible to adequately thank all those who have helped me with this book, giving generously of their time and talents. But perhaps I can at least indicate the size of my debts and publicly express my gratitude to those who have helped.
Foremost is Bradford Hardie, M.D., of El Paso, Texas, who translated a veritable stream of documents in German and read the galleys. His constant warm encouragement was like manna. My good friend Edward S. (Buddy) Miller of Malverne, New York, read many of the early chapters in manuscript and made extremely penetrating and valuable observations on them. Howard T. Oakley of Scotch Plains, New Jersey, and Kaljo Käärik, Ph.D., of Enskede, Sweden, read chapters, provided information, and exchanged views.
Many cryptologists or relatives of cryptologists took the time to talk with me or reply to my queries. I have acknowledged these debts in my notes, but I must pay special tribute to former Ambassador J. Rives Childs, who replied in detail to numerous questions and lent me his entire set of papers from his work in World War I; to Admiral Sir William James, who read the chapter on Room 40 and ransacked his voluminous memory for answers to many queries; to the late Yves Gyldén, who spent four days with me in Sweden; to Naotsune Watanabe and to Shiro Takagi, who wrote detailed reports of their World War II cryptanalytic experiences; to Dr. Hans Rohrbach, who set up some important appointments for me by long-distance telephone; to Harold R. Shaw, who wrote a 27-page reminiscence of his wartime work; to the Boris Hagelins, senior and junior, for hospitality and information; to Mrs. Malcolm Hay of Seaton, for information and photographs; to Parker Hitt, for an important memorandum and for the gift of his invaluable cipher papers; and to Mr. and Mrs. William F. Friedman for numerous kindnesses, though they steadfastly refused to discuss his government work, and for a gift made in 1947, upon my graduation from high school, that was a major step in my cryptologic education.
Many scholars very kindly replied to my queries about cryptology in their fields, and I have also acknowledged these in my notes. But especially generous were T. C. H. Raper of the India Office Library, London, who did a great deal of research on my behalf; C. E. Bosworth of St. Andrews University, Scotland, who furnished important background material in addition to a critical article; and Robert Wolfe, Philip Brower, and W. Neil Franklin of the National Archives, Washington, who replied with courtesy and dispatch to volleys of requests. Without the incredible resources of the New York Public Library and the courteous help of its staff in making them available, this book in its present form would not exist. A great deal of credit is due Mrs. Suzanne Oppenheimer, who typed the bulk of the book from execrable copy, and to Mrs. Harriet Simons, who typed the other chapters. Jenny Hauck made the photographic layouts. Geoffrey C. Jones of Lee-on-the-Solent, England, compiled the index, with some technical assistance by me.
The design department of The Macmillan Company and the Alden Press of Oxford, England, have overcome the many production problems to produce a very handsome book.
In a larger sense, I owe a great deal to former colleagues on Newsday, especially to Al Marlens, my former city editor, who taught me most of what I know about reporting and writing, and also to Bernie Bookbinder, who demonstrated that concern for the human must always be paramount; to Stan Isaacs, who showed how a subject can transcend itself; and to Stan Brooks, whose “Keep it light and bright!” galled me at the time but has since delivered me—I hope—from solemnity.
The errors are, of course, mine. If any reader cares to tell me of any corrections or additions, including personal reminiscences, I shall be very grateful to him.
DAVID KAHN
Windsor Gate
Great Neck, New York
Paris
A FEW WORDS
EVERY TRADE has its vocabulary. That of cryptology is simple, but even so a familiarity with its terms facilitates understanding. A glossary may also serve as a handy reference. The definitions in this one are informal and ostensive. Exceptions are ignored and the host of minor terms are not defined—the text covers these when they come up.
The plaintext is the message that will be put into secret form. Usually the plaintext is in the native tongue of the communicators. The message may be hidden in two basic ways. The methods of steganography conceal the very existence of the message. Among them are invisible inks and microdots and arrangements in which, for example, the first letter of each word in an apparently innocuous text spells out the real message. (When steganography is applied to electrical communications, such as a method that transmits a long radio message in a single short spurt, it is called transmission security.) The methods of cryptography, on the other hand, do not conceal the presence of a secret message but render it unintelligible to outsiders by various transformations of the plaintext.
Two basic transformations exist. In transposition, the letters of the plaintext are jumbled; their normal order is disarranged. To shuffle secret into ETCRSE is a transposition. In substitution, the letters of the plaintext are replaced by other letters, or by numbers or symbols. Thus secret might become 19 5 3 18 5 20, or XIWOXV in a more complicated system. In transposition, the letters retain their identities—the two e’s of secret are still present in ETCRSE—but they lose their positions, while in substitution the letters retain their positions but lose their identities. Transposition and substitution may be combined.
Substitution systems are much more diverse and important than transposition systems. They rest on the concept of the cipher alphabet. This is the list of equivalents used to transform the plaintext into the secret form. A sample cipher alphabet might be:
This graphically indicates that the letters of the plaintext are to be replaced by the cipher letters beneath them, and vice versa. Thus, enemy would become CHCME, and SWC would reduce to foe. A set of such correspondences is still called a “cipher alphabet” if the plaintext letters are in mixed order, or even if they are missing, because cipher letters always imply plaintext letters.
Sometimes such an alphabet will provide multiple substitutes for a letter. Thus plaintext e, for example, instead of always being replaced by, say, 16, will be replaced by any one of the figures 16, 74, 35, 21. These alternates are called homophones. Sometimes a cipher alphabet will include symbols that mean nothing and are intended to confuse interceptors; these are called nulls.
As long as only one cipher alphabet is in use, as above, the system is called monalphabetic. When, however, two or more cipher alphabets are employed in some kind of prearranged pattern, the system becomes polyalphabetic. A simple form of polyalphabetic substitution would be to add another cipher alphabet under the one given above and then to use the two in rotation, the first alphabet for the first plain
text letter, the second for the second, the first again for the third plaintext letter, the second for the fourth, and so on. Modern cipher machines produce polyalphabetic ciphers that employ millions of cipher alphabets.
Among the systems of substitution, code is distinguished from cipher. A code consists of thousands of words, phrases, letters, and syllables with the codewords or codenumbers (or, more generally, the codegroups) that replace these plaintext elements. A portion of a code might look like this:
codenumber plaintext
3964 emplacing
1563 employ
7260 en-
8808 enable
3043 enabled
0012 enabled to
This means, of course, that 0012 replaces enabled to. In a sense, a code comprises a gigantic cipher alphabet, in which the basic plaintext unit is the word or the phrase; syllables and letters are supplied mainly to spell out words not present in the code. In ciphers, on the other hand, the basic unit is the letter, sometimes the letter-pair (digraph or bigram), very rarely larger groups of letters (polygrams). The substitution and transposition systems illustrated above are ciphers. There is no sharp theoretical dividing line between codes and ciphers; the latter shade into the former as they grow larger. But in modern practice the differences are usually quite marked. Sometimes the two are distinguished by saying that ciphers operate on plaintext units of regular length (all single letters or all groups of, say, three letters), whereas codes operate on plaintext groups of variable length (words, phrases, individual letters, etc.). A more penetrating and useful distinction is that code operates on linguistic entities, dividing its raw material into meaningful elements like words and syllables, whereas cipher does not—cipher will split the t from the h in the, for example.
For 450 years, from about 1400 to about 1850, a system that was half a code and half a cipher dominated cryptography. It usually had a separate cipher alphabet with homophones and a codelike list of names, words, and syllables. This list, originally just of names, gave the system its name: nomenclator. Even though late in its life some nomenclators grew larger than some modern codes, such systems are still called “nomenclators” if they fall within this historical period. An odd characteristic is that nomenclators were always written on large folded sheets of paper, whereas modern codes are almost invariably in book or booklet form. The commercial code is a code used in business primarily to save on cable tolls; though some are compiled for private fims, many others are sold to the public and therefore provide no real secrecy.