Hillary Clinton's email problems began in her first days as secretary of state. She insisted on using her personal BlackBerry for all her email communications, but she wasn't allowed to take the device into her seventh-floor suite of offices, a secure space known as Mahogany Row.
For Clinton, this was frustrating. As a political heavyweight and chief of the nation's diplomatic corps, she needed to manage a torrent of email to stay connected to colleagues, friends and supporters. She hated having to put her BlackBerry into a lockbox before going into her own office.
Her aides and senior officials pushed to find a way to enable her to use the device in the secure area. But their efforts unsettled the diplomatic security bureau, which was worried that foreign intelligence services could hack her BlackBerry and transform it into a listening device.
On Feb. 17, 2009, less than a month into Clinton's tenure, the issue came to a head. Department security, intelligence and technology specialists, along with five officials from the National Security Agency, gathered in a Mahogany Row conference room. They explained the risks to Cheryl Mills, Clinton's chief of staff, while also seeking "mitigation options" that would accommodate Clinton's wishes.
"The issue here is one of personal comfort," one of the participants in that meeting, Donald Reid, the department's senior coordinator for security infrastructure, wrote afterward in an email that described Clinton's inner circle of advisers as "dedicated [BlackBerry] addicts."
Clinton used her BlackBerry as the group continued looking for a solution. But unknown to diplomatic security and technology officials at the department, there was another looming communications vulnerability: Clinton's BlackBerry was digitally tethered to a private email server in the basement of her family home, some 260 miles to the north in Chappaqua, N.Y., documents and interviews show.
Those officials took no steps to protect the server against intruders and spies, because they apparently were not told about it.
The vulnerability of Clinton's basement server is one of the key unanswered questions at the heart of a scandal that has dogged her campaign for the Democratic presidential nomination.
Since Clinton's private email account was brought to light a year ago in a New York Times report — followed by an Associated Press report revealing the existence of the server — the matter has been a source of nonstop national news. Private groups have filed lawsuits under the Freedom of Information Act. Investigations were begun by congressional committees and inspector general's offices in the State Department and the U.S. Intelligence Community, which referred the case to the FBI in July for "counterintelligence purposes" after determining that the server carried classified material.
The FBI is now trying to determine whether a crime was committed in the handling of that classified material. It is also examining whether the server was hacked.
One hundred forty-seven FBI agents have been deployed to run down leads, according to a lawmaker briefed by FBI Director James B. Comey. The FBI has accelerated the investigation because officials want to avoid the possibility of announcing any action too close to the election.
The Washington Post reviewed hundreds of documents and interviewed more than a dozen knowledgeable government officials to understand the decisions and the implications of Clinton's actions. The resulting scandal revolves around questions about classified information, the preservation of government records and the security of her email communication.
From the earliest days, Clinton aides and senior officials focused intently on accommodating the secretary's desire to use her private email account, documents and interviews show.
Throughout, they paid insufficient attention to laws and regulations governing the handling of classified material and the preservation of government records, interviews and documents show. They also neglected repeated warnings about the security of the BlackBerry while Clinton and her closest aides took obvious security risks in using the basement server.
Senior officials who helped Clinton with her BlackBerry claim they did not know details of the basement server, the State Department said, even though they received emails from her private account. One email written by a senior official mentioned the server.
The scandal has pitted those who say Clinton was innocently trying to find the easiest way to communicate against those who say she placed herself above the law in a quest for control of her records. She and her campaign have been accused of confusing matters with contradictory and evolving statements that minimized the consequences of her actions.
Clinton, 68, declined to be interviewed. She has said repeatedly that her use of the private server was benign and that there is no evidence of any intrusion.
In a news conference last March, she said: "I opted for convenience to use my personal email account, which was allowed by the State Department, because I thought it would be easier to carry just one device for my work and for my personal emails instead of two."
During a Democratic debate on March 9, she acknowledged using poor judgment but maintained she was permitted to use her own server: "It wasn't the best choice. I made a mistake. It was not prohibited. It was not in any way disallowed."
The unfolding story of Clinton's basement server has outraged advocates of government transparency and mystified political supporters and adversaries alike. Judge Emmet G. Sullivan of the U.S. District Court in Washington, D.C., who is presiding over one of the FOIA lawsuits, has expressed puzzlement over the affair. He noted that Clinton put the State Department in the position of having to ask her to return thousands of government records — her work email.
"Am I missing something?" Sullivan asked during a Feb. 23 hearing. "How in the world could this happen?"
Hillary Clinton began preparing to use the private basement server after President Obama picked her to be his secretary of state in November 2008. The system was already in place. It had been set up for former president Bill Clinton, who used it for personal and Clinton Foundation business.
On Jan. 13, 2009, a longtime aide to Bill Clinton registered a private email domain for Hillary Clinton, clintonemail.com, that would allow her to send and receive email through the server.
Eight days later, she was sworn in as secretary of state. Among the multitude of challenges she faced was how to integrate email into her State Department routines. Because Clinton did not use desktop computers, she relied on her personal BlackBerry, which she had started using three years earlier.
For years, employees across the government had used official and private email accounts.
The new president was making broad promises about government transparency that had a bearing on Clinton's communication choices. In memos to his agency chiefs, Obama said his administration would promote accountability through the disclosure of a wide array of information, one part of a "profound national commitment to ensuring an open government." That included work emails.
One year earlier, during her own presidential campaign, Clinton had said that if elected, "we will adopt a presumption of openness and Freedom of Information Act requests and urge agencies to release information quickly."
But in those first few days, Clinton's senior advisers were already taking steps that would help her circumvent those high-flown words, according to a chain of internal State Department emails released to Judicial Watch, a conservative nonprofit organization suing the government over Clinton's emails.
Leading that effort was Mills, Clinton's chief of staff. She was joined by Clinton adviser Huma Abedin, Undersecretary Patrick Kennedy and Lewis Lukens, a senior career official who served as Clinton's logistics chief. Their focus was on accommodating Clinton.
Mills wondered whether the department could get her an encrypted device like the one from the NSA that Obama used.
"If so, how can we get her one?" Mills wrote the group on Saturday evening, Jan. 24.
Lukens responded that same evening, saying he could help set up "a stand alone PC in the Secretary's office, connected to the internet (but not through our system) to enable her to check her emails from her desk."
Kennedy wrote that a "stand-alone separate network PC" was a "great idea."
Abedin and Mills declined to comment for this article, according to Clinton spokesman Brian Fallon. Lukens also declined to comment, according to the State Department.
As undersecretary for management, Kennedy occupies a central role in Clinton's email saga. The department acknowledged that Kennedy, as part of his normal duties, helped Clinton with her BlackBerry. But in a statement, the department said: "Under Secretary Kennedy maintains that he was unaware of the email server. Completely separate from that issue, Under Secretary Kennedy was aware that at the beginning of her tenure, Secretary Clinton's staff was interested in setting up a computer at the Department so she could email her family during the work day.
"As we have previously made clear — no such computer was ever set up. Furthermore, Under Secretary Kennedy had very little insight into Secretary Clinton's email practices including how frequently or infrequently then-Secretary Clinton used email."
As it happened, Clinton would never have a government BlackBerry, personal computer or email account. A request for a secure device from the NSA was rebuffed at the outset: "The current state of the art is not too user friendly, has no infrastructure at State, and is very expensive," Reid, the security official, wrote in an email on Feb. 13, adding that "each time we asked the question 'What was the solution for POTUS?' we were politely told to shut up and color."
Clinton would continue to use her BlackBerry for virtually all of her government communication, but not on Mahogany Row.
Her first known BlackBerry communication through the basement server came on Jan. 28, 2009, when Clinton exchanged notes with Army Gen. David H. Petraeus, then chief of the U.S. Central Command, according to a State Department spokeswoman. It has not been released.
Few knew the details behind the new clintonemail.com address. But news about her choice to use her own BlackBerry spread quickly among the department's diplomatic security and "intelligence countermeasures" specialists.
Their fears focused on the seventh floor, which a decade earlier had been the target of Russian spies who managed to plant a listening device inside a decorative chair-rail molding not far from Mahogany Row. In more recent years, in a series of widely publicized cyberattacks, hackers breached computers at the department along with those at other federal agencies and several major corporations.
The State Department security officials were distressed about the possibility that Clinton's BlackBerry could be compromised and used for eavesdropping, documents and interviews show.
After the meeting on Feb. 17 with Mills, security officials in the department crafted a memo about the risks. And among themselves, they expressed concern that other department employees would follow the "bad example" and seek to use insecure BlackBerrys themselves, emails show.
As they worked on the memo, they were aware of a speech delivered by Joel F. Brenner, then chief of counterintelligence at the Office of the Director of National Intelligence, on Feb. 24 at a hotel in Vienna, Va., a State Department document shows. Brenner urged his audience to consider what could have happened to them during a visit to the recent Beijing Olympics.
"Your phone or BlackBerry could have been tagged, tracked, monitored and exploited between your disembarking the airplane and reaching the taxi stand at the airport," Brenner said. "And when you emailed back home, some or all of the malware may have migrated to your home server. This is not hypothetical."
At the time, Clinton had just returned from an official trip that took her to China and elsewhere in Asia. She was embarking on another foray to the Middle East and Europe. She took her BlackBerry with her.
In early March, Assistant Secretary for Diplomatic Security Eric Boswell delivered a memo with the subject line "Use of Blackberries in Mahogany Row."
"Our review reaffirms our belief that the vulnerabilities and risks associated with the use of Blackberries in the Mahogany Row [redacted] considerably outweigh the convenience their use can add," the memo said.
He emphasized: "Any unclassified Blackberry is highly vulnerable in any setting to remotely and covertly monitoring conversations, retrieving e-mails, and exploiting calendars."
Nine days later, Clinton told Boswell that she had read his memo and "gets it," according to an email sent by a senior diplomatic security official. "Her attention was drawn to the sentence that indicates (Diplomatic Security) have intelligence concerning this vulnerability during her recent trip to Asia," the email said.
But Clinton kept using her private BlackBerry — and the basement server.
The server was nothing remarkable, the kind of system often used by small businesses, according to people familiar with its configuration at the end of her tenure. It consisted of two off-the-shelf server computers. Both were equipped with antivirus software. They were linked by cable to a local Internet service provider. A firewall was used as protection against hackers.
Few could have known it, but the email system operated in those first two months without the standard encryption generally used on the Internet to protect communication, according to an independent analysis that Venafi Inc., a cybersecurity firm that specializes in the encryption process, took upon itself to publish on its website after the scandal broke.
Not until March 29, 2009 — two months after Clinton began using it — did the server receive a "digital certificate" that protected communication over the Internet through encryption, according to Venafi's analysis.
It is unknown whether the system had some other way to encrypt the email traffic at the time. Without encryption — a process that scrambles communication for anyone without the correct key — email, attachments and passwords are transmitted in plain text.
"That means that anyone could have accessed it. Anyone," Kevin Bocek, vice president of threat intelligence at Venafi, told The Post.
The system had other features that made it vulnerable to talented hackers, including a software program that enabled users to log on directly from the World Wide Web.
Four computer-security specialists interviewed by The Post said that such a system could be made reasonably secure but that it would need constant monitoring by people trained to look for irregularities in the server's logs.
"For data of this sensitivity . . . we would need at a minimum a small team to do monitoring and hardening," said Jason Fossen, a computer-security specialist at the SANS Institute, which provides cybersecurity training around the world.
The man Clinton has said maintained and monitored her server was Bryan Pagliano, who had worked as the technology chief for her political action committee and her presidential campaign. It is not clear whether he had any help. Pagliano had also provided computer services to the Clinton family. In 2008, he received more than $5,000 for that work, according to financial disclosure statements he filed with the government.
In May 2009, with Kennedy's help, Pagliano landed a job as a political employee in the State Department's IT division, documents and interviews show. It was an unusual arrangement.
At the same time, Pagliano apparently agreed to maintain the basement server. Officials in the IT division have told investigators they could not recall previously hiring a political appointee. Three of Pagliano's supervisors also told investigators they had no idea that Clinton used the basement server or that Pagliano was moonlighting on it.
Through an attorney, Pagliano declined a request from The Post for an interview. He also refused a request from the Senate Judiciary and Homeland Security and Governmental Affairs committees to discuss his role. On Sept. 1, 2015, his attorney told the committees that he would invoke his Fifth Amendment rights if any attempt was made to compel his testimony. He was later given immunity by the Justice Department in exchange for his cooperation, according to articles in the New York Times and The Post.
In a statement, Clinton's campaign said the server was protected but declined to provide technical details. Clinton officials have said that server logs given to authorities show no signs of hacking.
"The security and integrity of her family's electronic communications was taken seriously from the onset when it was first set up for President Clinton's team," the statement said. "Suffice it to say, robust protections were put in place and additional upgrades and techniques employed over time as they became available, including consulting and employing third party experts."
The statement added that "there is no evidence there was ever a breach."
The number of emails moving through the basement system increased quickly as Hillary Clinton dove into the endless details of her globetrotting job. There were 62,320 in all, an average of 296 a week, nearly 1,300 a month, according to numbers Clinton later reported to the State Department. About half of them were work-related.
Her most frequent correspondent was Mills, her chief of staff, who sent thousands of notes. Next came Abedin, the deputy chief of staff, and Jacob Sullivan, also a deputy chief of staff, according to a tally by The Post.
The majority went to two different addresses that Clinton sometimes used interchangeably on a single chain of email, hdr22@clintonemail.com and hrod17@clintonemail.com, making it immediately apparent that the emails were not coming from or going to a government address.
Most of her emails were routine, including those sent to friends. Some involved the coordination of efforts to bring aid to Haiti by the State Department and her husband's New York-based Clinton Foundation — notes that mixed government and family business, the emails show.
Others involved classified matters. State Department and Intelligence Community officials have determined that 2,093 email chains contained classified information. Most of the classified emails have been labeled as "confidential," the lowest level of classification. Clinton herself authored 104 emails that contained classified material, a Post analysis later found.
Before the server received a digital certificate marking the use of standard encryption, Clinton and her aides exchanged notes touching on North Korea, Mexico, Afghanistan, military advisers, CIA operations and a briefing for Obama.
Clinton adviser Philippe Reines wrote a note to her about Afghanistan President Hamid Karzai. Reines started his note by reminding Clinton that Reines's "close friend Jeremy Bash is now [CIA Director Leon E.] Panetta's Chief of Staff." The rest of the note was redacted before release, under grounds that it was national-security-sensitive.
On Sunday, March 29, 2009, just hours before standard encryption on the server began, Sullivan emailed Clinton a draft of a confidential report she was to make to Obama. "Attached is a draft of your Mexico trip report to POTUS," Sullivan wrote.
In the high-pressure world of diplomacy, the sharing of such material had been a discreet but common practice for many years. Officials who manage problems around the clock require a never-ending flow of incisive information to make timely decisions.
Not all classified material is equally sensitive. Much of it involves discussions about foreign countries or leaders, not intelligence sources and methods. Working with classified materials can be cumbersome and, in the case of low-level classification, annoying.
On Feb. 10, 2010, in an exchange with Sullivan, Clinton vented her frustration one day when she wanted to read a statement regarding José Miguel Insulza, then secretary general of the Organization of American States. Sullivan wrote that he could not send it to her immediately because the department had put it on the classified network.
"It's a public statement! Just email it," Clinton shot back, just moments later.
"Trust me, I share your exasperation," Sullivan wrote. "But until ops converts it to the unclassified email system, there is no physical way for me to email it. I can't even access it."
Early on June 17, 2011, Clinton grew impatient as she waited for "talking points" about a sensitive matter that had to be delivered via a secure line.
"They say they've had issues sending secure fax. They're working on it," Sullivan wrote his boss.
Clinton told him to take a shortcut.
"If they can't, turn into nonpaper w no identifying heading and send nonsecure," she said.
Clinton spokesman Fallon said she was not trying to circumvent the classification system.
"What she was asking was that any information that could be transmitted on the unclassified system be transmitted," he said. "It is wrong to suggest that she was requesting otherwise. The State Department looked into this and confirmed that no classified material was sent through a non-secure fax or email."
Security remained a constant concern. On June 28, 2011, in response to reports that Gmail accounts of government workers had been targeted by "online adversaries," a note went out over Clinton's name urging department employees to "avoid conducting official Department business from your personal email accounts."
But she herself ignored the warning and continued using her BlackBerry and the basement server.
In December 2012, near the end of Clinton's tenure, a nonprofit group called Citizens for Responsibility and Ethics in Washington, or CREW, filed a FOIA request seeking records about her email. CREW received a response in May 2013: "no records responsive to your request were located."
Other requests for Clinton records met the same fate — until the State Department received a demand from the newly formed House Select Committee on Benghazi in July 2014. The committee wanted Clinton's email, among other things, to see what she and others knew about the deadly attack in Libya and the response by the U.S. government.
Officials in the department's congressional affairs office found some Clinton email and saw that she had relied on the private domain, not the department's system.
Secretary of State John F. Kerry resolved to round up the Clinton emails and deliver them to Congress as quickly as possible. Department officials reached out to Clinton informally in the summer of 2014. On Oct. 28, 2014, the department contacted Clinton and the offices of three other former secretaries — Madeleine K. Albright, Condoleezza Rice and Colin L. Powell — asking if they had any email or other federal records in their possession.
Albright and Rice said they did not use email while at State. Powell, secretary of state from 2001 to 2005, had a private email account through America Online but did not retain copies of his emails. The inspector general for the State Department found that Powell's personal email account had received two emails from staff that contained "national security information classified at the Secret or Confidential levels."
Clinton lawyer David Kendall later told the State Department that her "use of personal email was consistent with the practices of other Secretaries of State," citing Powell in particular, according to a letter he wrote in August.
But Powell's circumstances also differed from Clinton's in notable ways. Powell had a phone line installed in his office solely to link to his private account, which he generally used for personal or non-classified communication. At the time, he was pushing the department to embrace the Internet era and wanted to set an example.
"I performed a little test whenever I visited an embassy: I'd dive into the first open office I could find (sometimes it was the ambassador's office). If the computer was on, I'd try to get into my private email account," Powell wrote in "It Worked for Me: In Life and Leadership.""If I could, they passed."
Powell conducted virtually all of his classified communications on paper or over a State Department computer installed on his desk that was reserved for classified information, according to interviews. Clinton never had such a desktop or a classified email account, according to the State Department.
On Dec. 5, 2014, Clinton lawyers delivered 12 file boxes filled with printed paper containing more than 30,000 emails. Clinton withheld almost 32,000 emails deemed to be of a personal nature.
The department began releasing the emails last May, starting with some 296 emails requested by the Benghazi committee. In reviewing those emails, intelligence officials realized that some contained classified material.
Clinton and her campaign have offered various responses to questions about the classifications. At first, she flat-out denied that her server ever held any. "There is no classified material," she said at a March 10, 2015, news conference.
Her campaign later released a statement saying she could not have known whether material was classified, because it was not labeled as such. "No information in Clinton's emails was marked classified at the time she sent or received them," the statement said.
Clinton has also suggested that many of the emails were classified as a formality only because they were being prepared for release under a FOIA request. Her campaign has said that much of the classified material — in emails sent by more than 300 individuals — came from newspaper accounts and other public sources.
"What you are talking about is retroactive classification," she said during a recent debate. "And I think what we have got here is a case of overclassification." Her statement appears to conflict with a report to Congress last year by inspectors general from the State Department and the group of spy agencies known as the Intelligence Community. They made their report after the discovery that four emails, from a sample of 40 that went through her server, contained classified information.
"These emails were not retroactively classified by the State Department," the report said. "Rather these emails contained classified information when they were generated and, according to IC classification officials, that information remains classified today. This classified information should never have been transmitted via an unclassified personal system."
One of those four emails has since been declassified and released publicly by the State Department. The department has questioned the classification of another of those emails.
Twenty-two emails discovered later were deemed so highly classified that they were withheld in their entirety from public release. "They are on their face sensitive and obviously classified," Rep. Chris Stewart (R-Utah), a member of the House Permanent Select Committee on Intelligence, told The Post. "This information should have been maintained in the most secure, classified, top-secret servers."
Fallon pointed out that none of those emails originated with Clinton, something that he said Dianne Feinstein (D-Calif.), the Senate Select Intelligence Committee vice chairman, has noted. "We strongly disagree with the decision to withhold these emails in full," he said.
Under Title 18, Section 1924, of federal law, it is a misdemeanor punishable by fines and imprisonment for a federal employee to knowingly remove classified information "without authority and with the intent to retain such documents or materials at an unauthorized location."
Previous cases brought under the law have required proof of an intent to mishandle classified information, a high hurdle in the Clinton case.
The basement server also put Clinton at risk of violating laws and regulations aimed at protecting and preserving government records.
In a statement, Clinton's campaign said she had received "guidance regarding the need to preserve federal records" and followed those rules. "It was her practice to email government employees on their '.gov' email address. That way, work emails would be immediately captured and preserved in government record-keeping systems," the statement said.
Fallon said that "over 90 percent" of the more than 30,000 work-related emails "were to or from government email accounts."
Specialists interviewed by The Post said her practices fell short of what laws and regulations mandated. Some of those obligations were spelled out a few months before Clinton took office in National Archives and Records Administration Bulletin 2008-05, which said every email system was supposed to "permit easy and timely retrieval" of the records.
The secretary of state's work emails are supposed to be preserved permanently. In addition, rules also mandated that permanent records are to be sent to the department's Records Service Center "at the end of the Secretary's tenure or sooner if necessary" for safekeeping.
Under Title 18, Section 2071, it is a misdemeanor to take federal records without authorization, something that is sometimes referred to as the "alienation" of records. The law is rarely enforced, but a conviction can carry a fine or imprisonment.
Jason R. Baron, a former director of litigation at the National Archives and Records Administration, told the Senate Judiciary Committee last year he believed that Clinton's server ran afoul of the rules. In a memo to the committee, Baron wrote that "the setting up of and maintaining a private email network as the sole means to conduct official business by email, coupled with the failure to timely return email records into government custody, amounts to actions plainly inconsistent with the federal recordkeeping laws."
On May 19, 2015, in response to a FOIA lawsuit from the media organization Vice News, U.S. District Judge Rudolph Contreras ordered all the email to be released in stages, with redactions.
One notable email was sent in August 2011. Stephen Mull, then serving as the department's executive secretary, emailed Abedin, Mills and Kennedy about getting a government-issued BlackBerry linked to a government server for Clinton.
"We are working to provide the Secretary per her request a Department issued Blackberry to replace personal unit, which is malfunctioning (possibly because of her personal email server is down.) We will prepare two version for her to use — one with an operating State Department email account (which would mask her identity, but which would also be subject to FOIA requests)."
Abedin responded decisively.
"Steve — let's discuss the state blackberry. doesn't make a whole lot of sense."
Fallon said the email showed that the secretary's staff "opposed the idea of her identity being masked."
Last month, in a hearing about a Judicial Watch lawsuit, U.S. District Judge Sullivan cited that email as part of the reason he ordered the State Department produce records related to its initial failures in the FOIA searches for Clinton's records.
Speaking in open court, Sullivan said legitimate questions have been raised about whether Clinton's staff was trying to help her to sidestep FOIA.
"We're talking about a Cabinet-level official who was accommodated by the government for reasons unknown to the public. And I think that's a fair statement: For reasons heretofore unknown to the public. And all the public can do is speculate," he said, adding: "This is all about the public's right to know."
Alice Crites contributed to this report.
This article was written by Robert O’Harrow Jr. from The Washington Post and was legally licensed through the NewsCred publisher network.
Join the conversation about this story »
NOW WATCH: Ancient Romans had perfect teeth because their diets were low in one substance