Buzzfeed news posted HERE about how the Veris Group was contracted to penetration test Palantir’s network. Palantir handles extremely sensitive data so they have what many would consider a very secure network. The reason I found this article interesting is there are many organizations that invest a ton in security and “assume” they are secure. It is great to see that Palantir put their security to the test and is sharing that they found holes, which are being secured. Kudos to them. Hopefully others will follow the same strategy by actually evaluating their security once they feel it is up to par.
NOTE: A penetration test should be conducted AFTER you believe you have properly hardened your network. Doing so prior is a waste of money.
Palantir Technologies has cultivated a reputation as perhaps the most formidable data analysis firm in Silicon Valley, doing secretive work for defense and intelligence agencies as well as Wall Street giants. But when Palantir hired professional hackers to test the security of its own information systems late last year, the hackers found gaping holes that left data about customers exposed.
Palantir, valued at $20 billion, prides itself on an ability to guard important secrets, both its own and those entrusted to it by clients. But after being brought in to try to infiltrate these digital defenses, the cybersecurity firm Veris Group concluded that even a low-level breach would allow hackers to gain wide-ranging and privileged access to the Palantir network, likely leading to the “compromise of critical systems and sensitive data, including customer-specific information.”
This conclusion was presented in a confidential report, reviewed by BuzzFeed News, that detailed the results of a hacking exercise run by Veris over three weeks in September and October last year. The report, submitted on October 19, has been closely guarded inside Palantir and is described publicly here for the first time. “Palantir Use Only” is plastered across each page.
It is not known whether Palantir’s systems have ever been breached by real-world intruders. But the results of the hacking exercise — known as a “red team” test — show how a company widely thought to have superlative ability to safeguard data has struggled with its own data security.
The red team intruders, finding that Palantir lacked crucial internal defenses, ultimately “had complete control of PAL’s domain,” the Veris report says, using an acronym for Palantir. The report recommended that Palantir “immediately” take specific steps to improve its data security.
“The findings from the October 2015 report are old and have long since been resolved,” Lisa Gordon, a Palantir spokesperson, said in an emailed statement. “Our systems and our customers’ information were never at risk. As part of our best practices, we conduct regular reviews and tests of our systems, like every other technology company does.”
Virtually every company is vulnerable to hacks, to varying degrees. In recent years, red teams generally have had a high success rate in getting deep inside of companies’ networks, and they virtually always find at least some security flaws, according to an industry source. That Palantir did a red team exercise shows that it wanted to identify and repair any such flaws. The Veris report notes multiple strengths in Palantir’s defenses, including an “excellent” response by its security staff.
“Regular red team testing is the industry standard of excellence in maintaining a proactive security posture,” David McGuire, the director of Veris’ adaptive threat division, which handles red team services, said in an emailed statement. “Since the red team exercise conducted in 2015, Palantir has consistently carried out similar exercises with Veris Group and other vendors on a regular basis.”
Veris, a cybersecurity services and consulting firm based near Washington, DC, works with customers including Microsoft, AT&T, and the Department of Justice, according to its website. For Palantir, Veris staff acted as hackers to find out whether Palantir’s cybersecurity team could detect and stop them.
The exercise was not meant to test whether Veris could breach Palantir’s external wall. Instead, the red team was deliberately let in, to simulate what would happen if a Palantir employee succumbed to a very common and highly effective break-in technique called “spear phishing” (in which staff are targeted with innocuous-seeming emails containing harmful links or files that give attackers access to a computer). But from that point on, the Veris team went into hacker mode, using a range of tricks to spread through Palantir’s cyber fortress, the report shows.
That fortress turned out to have major vulnerabilities, and the Veris intruders soon sat themselves on the throne. In what the report calls a “complete compromise,” the intruders uncovered encryption keys and administrative credentials that allowed them to travel widely inside the network, accessing source code, office surveillance footage, and the internal wiki, which held sensitive data about customers and projects, according to the report.
Beyond these secrets, the red team intruders accessed Palantir’s network equipment, which would have let them control the company’s internet connection if they so chose. They even found what appeared to be “access to customer infrastructure,” according to the report, or hardware powering customers’ information technology. The report says that any hacker who got this far would “possibly” be able to hack Palantir’s customers as well.
Repeatedly, the red team intruders followed a straightforward process: Find credentials for a high-level account, and then use those credentials to ferret out additional credentials that conferred even more access. They were able to “position themselves in the network for long-term persistence,” the report says.
In a sign of their deep access, the intruders created a software tunnel to smuggle data out to their own servers, without being detected for most of the exercise, according to the report. Their presence was finally discovered, the report says, after they broke into the laptops of information security employees — but even then, the intruders were able to monitor the employees’ countermoves in real time, shifting tactics to evade them.
Palantir wasn’t totally defenseless, the report shows. Its network was segmented in a way that initially prevented the Veris intruders from moving very far, forcing them to take a riskier approach that increased their chances of being detected — though they managed to slip through without setting off any alarms. The company also made use of two-factor authentication, which at first “severely hampered” the intruders’ plans but ultimately just forced them, again, to use a more conspicuous strategy to gain access, according to the report.
When Palantir’s information security employees finally discovered the intruders, they “provided a rapid network response in which they identified and mitigated” the “majority” of the red team’s actions within days, the report says. Compared with other large companies, this defensive response was unusually robust, the industry source said, based on a reading of the report.
Started in part with CIA money, the 12-year-old Palantir has developed an aura of secrecy and potency that helps it recruit bright engineers and attract corporate clients. Its chairman is Peter Thiel, the widely admired venture capitalist and former PayPal CEO (who recently admitted to secretly funding a lawsuit brought by the wrestler Hulk Hogan against Gawker Media). Part software shop and part consulting firm, Palantir places its “forward deployed engineers” on-site at client offices and uses custom-tailored software to crunch vast amounts of data.
Its customers include financial institutions, such as the giant hedge fund Bridgewater Associates, and government groups such as the military’s Special Operations Command. Palantir is the third most valuable American technology startup, behind only Uber and Airbnb.
At the same time, Palantir has recently lost blue-chip clients, has struggled to stem staff departures, and has recorded 2015 revenue that was less than a quarter of its customer bookings, according to a BuzzFeed News report in early May. The report, based on a trove of internal documents and insider interviews, revealed that 102 employees had left Palantir this year through mid-April, or 5.8% of all staff.
When it comes to cybersecurity, experts advise companies to fortify their internal defenses — to ensure an initial breach doesn’t become a total takeover. Hackers are so good at getting through the external wall, often using spear phishing, that cyber experts routinely just assume such attackers will get in, according to Anup Ghosh, CEO of cyber threat firm Invincea.
“Almost every breach you read about happens through spear phishing, and the weak link is the human behind the keyboard. Spear phishing always, always works. You can’t un-train human behavior,” Ghosh told BuzzFeed News. “How do we make it so that these attacks can’t compromise the whole computer?”
As of last fall, Palantir had an inadequate answer to that question, the Veris report shows.
When the red team intruders from Veris got inside, they found that standard user accounts had local administrative access — rendering Palantir more vulnerable. This setup “effectively granted administrative access to the red team” and “removed a major hurdle in the attack methodology,” the report says. In general, tech companies tend to give more control to employees than more traditional companies do. For Palantir, allowing low-level users to have high-level access was a “high” risk, Veris concluded.
“Administrative privileges should be granted explicitly and only when necessary,” Veris says in the report, urging Palantir to “remove standard domain users from the local administrators group or implement controls to delegate administrative permissions as necessary.”
The red team soon found that a local administrative account — with an easily identifiable name — was enabled on numerous computers in the network, with identical password hashes on each computer, the report says. A password hash is a way of obscuring a password in a hard-to-crack format.
But the red team didn’t need to crack the hashes. Since they were already inside, they could use a technique called “pass-the-hash” to feed hashes, rather than the underlying passwords, into password verification systems, allowing them to hop from computer to computer, the report shows.
(Pass-the-hash attacks are a widely known way of exploiting a vulnerability in Windows systems, and Microsoft has released security updates to mitigate the problem. “But ultimately, all we’re doing is we’re in an arms race with the hackers,” Jonathan Cogley, founder of security software company Thycotic, said in a presentation on pass-the-hashlast year.)
Veris classified the riskiness of the pass-the-hash vulnerability as “high,” recommending that Palantir disable the local administrative account where possible and use unique passwords for each computer.
The red team had difficulty, however, moving outside its network segment, analogous to a walled room inside a building. So the team infiltrated a terminal server — a central server where multiple people, including some with privileged access, log on and perform important tasks. From this new vantage point, the intruders scanned the surrounding network and found credentials for a domain administrator account, which conferred a high level of access, the report shows.
Terminal servers make an obvious target for hackers, since they often contain high-level credentials. They tend to be well protected, however, making a hack risky. In Palantir’s case, the red team found that logon activities at the terminal server were “not heavily monitored,” according to the report.
After scooping up credentials for a system engineer, the intruders broke into systems related to the proxy server, an important data hub. They then set up an encrypted tunnel running outside the network to their own servers, for pilfering data. This step, again, would be risky for a hacker. But the tunnel “went undetected for most of the engagement,” allowing the red team to “access and data-mine internal Palantir web applications, as well as access servers of interest,” the report says.
“The lack of egress controls can allow an attacker to establish unrestricted communications with a remote server, outside of Palantir’s network,” the Veris report says. “An attacker can also leverage this vulnerability to successfully exfiltrate sensitive data from Palantir’s systems.”
Before long, the red team had found the central wiki, where they “observed sensitive data pertaining to customers, budgets, deployments, and locations,” according to the report. Palantir uses quirky codenames to refer to its customers — as of last month, “Nancy Drew” was Nasdaq, and “Stones” was BP, for example — and the red team was in some cases “able to map codenames to customers,” the report says. In a separate application, the intruders found “source code for a number of sensitive projects.”
The red team’s next target was a secure database — essentially a safe — that stored the credentials to access critically important systems. A master key, itself stored in a secure file, would open the safe.
That the red team even found this safe at all is a concern, the report suggests. Several “essential information systems,” including the safe, were “relatively easy to locate and access on the domain,” according to the report.
After analyzing the master key file, the intruders were able to decrypt it, opening the safe, the report shows.
Using information they found there, the intruders accessed switches and other devices that underpinned communication on the network. Anyone with access to a company’s network equipment can control the flow of network traffic — with the ability to filter traffic or even reroute it — though there is no indication the red team attempted to do this.
In addition, “access to customer infrastructure appeared to be stored” in the safe, according to the report. In enterprise computing, “infrastructure” is a broad term that includes the servers, routers, and other pieces of equipment that a company relies on for its business.
A hacker, moreover, could exploit weaknesses in the safe’s security “to access credentials and valuable information that will ultimately lead to compromise of most, if not all, of Palantir’s network devices, systems, and possibly customer infrastructure as well,” the report says. Veris urged Palantir to add another layer of security to the file containing the master key.
McGuire of Veris said in a phone interview with BuzzFeed News that, in general, a red team would never do anything “destructive” during an exercise, nor would it ever “test organizations that are not signed up for the assessment.” He said: “The demonstration of access is as far as we go.”
Even Palantir’s defense efforts were visible to the red team. The intruders found an “InfoSec Onboarding” page on the wiki that detailed Palantir’s security infrastructure. They monitored security devices and “ensured that their actions were not being logged.”
This was when, according to the report, the red team intruders had “complete control” of the Palantir domain. Their final task was to break into the Mac laptops of information security employees — the fortress guards. This they did, using a system that typically sent out software updates, and soon were able to get passwords and screenshots, review saved files, and “observe all user activity,” the report says.
They were finally caught while attempting to upload a screenshot to one of their own servers, according to the report. A piece of security software called Little Snitch — which regulates data sent out from a computer to the internet — was installed on one of the information security employees’ laptops, and it flagged the suspicious upload attempt, the report says. Little Snitch, while popular in the cybersecurity world, was not standard software for these employees, according to one person familiar with the matter.
Soon, Palantir security employees identified the red team’s attack tools and set up firewalls to block communications to the red team servers. These defenders “successfully demonstrated the ability to trace malicious activity across the domain and take the appropriate steps to neutralize an insider threat,” the report says.
But the red team still had an edge.
“The assessment team was able to observe all investigative actions as progress was tracked and noted,” the Veris report says. This allowed the intruders to “maintain their presence in the network, even after discovery,” by changing key elements of their attack tools.
According to the Veris report, “the red team successfully evaded defenders up until the last day of the engagement.”
Sheera Frenkel contributed to this report.
Thanks for the post! I’m working on my website providing Penetration Testing and Website Security