Uber's Scraping Program that Collected Massive Amount of Data from Competitors

One of Uber’s main division was their Marketplace Analytics team (MA), which is a nice way of saying corporate espionage and competitor intelligence gathering team. Through MA, Uber managed to scrape data from hundreds of companies, collecting millions of records relating to technology, executives, and drivers. To complement this division, Uber’s Strategic Services Group (SSG) used more direct intervention methods to gather and secure data for “security” purposes. The distinction between the two groups was defined as MA deals with international competition information while SSG deals with protecting Uber personnel, drivers and collecting information about protestors and other threats to Uber.

While there is no definitive proof that either MA or SSG violated any national or international laws, since they claim that all the data was drawn from accessible public sites on the internet, their activites have been put under the spotlight in the recent Waymo-Uber court case, where industrial espionage and trade secret theft is the main theme of the case of Waymo against Uber. To bolster Waymo's claims of illegal activities performed by Uber, former Uber employee Richard Jacobs, wrote a 37-page testimony that claims Uber used their MA and SSG to steal confidential information from competitors. This important and possibly vital piece of evidence was introduced into the trial when the ATG personally sent the Judge a copy of the letter. The fact that Uber did not present the letter during the trial and the fact that the ATG considered it important enough to interfere in a trial (something that is rarely done) makes the letter central to the case.

Due to the sudden and unexpected appearance of the new evidence, the Judge presiding over the case delayed it from November to February 2018, so he and the prosecution could go over the new evidence. The letter points out that the MA and SSG used digital devices that would not allow their activity to be traced back to Uber. They used transient messaging apps and human surveillance to retrieve confidential information from their competitors. The letter accuses Uber of trying to hide their actions from everyone including their competitors and if the situation arises, from law enforcement and the courts.

How to Build a CIA and FBI

The MA originated within Uber as Competitive Intelligence (COIN). This unit initially set up disassociated servers to hold all the data they collected on their competition and developed and used a program called Hell. Hell was a program that would track Lyft drivers and try to persuade them to work for Uber. The program was capable of retrieving the Lyft drivers ID number and thereby track their locations via GPS. The Amazon server that COIN used was registered to an Uber security engineer in March 2015 and was closed in 2016. During that time, COIN started to transfer their intelligence gathering systems under a new infrastructure and name. The COIN team continued to expand and eventually all the units it held merged into one team, the Marketing Analytics team. This was a dedicated team of professional data scientists and software engineers. Amongst the team, you could count former government employees, and at that time, their focus shifted from local and national intelligence gathering to international field work.

The newly merged MA started to surveil overseas competitors including India's Ola and China's Didi Chuxing. The surveillance on Didi stopped once Uber sold its Chinese presence to Didi for $35 billion and got a seat on the board. At the same time that MA was gathering intel on Uber's competitors, SSG hired out overseas work to contractors who would find and accumulate physical evidence on threats from anti-Uber groups as well as from competitors, such as recording conversations that were made between Grab and Didi executives. This was all mentioned in Jacobs letter.

The two teams would store their information on the detached secured server and were given computers to use only for this operation. All connection to Uber was destroyed so that the operation could go ahead without leading back to Uber. The team was also taught how to use Wickr messaging which would encrypt the message and then delete the message after it was sent, and as would not leave any traces. Added to all this covert framework, was the complete disassociation of the servers to Uber, so that they would not be targeted by hackers and crackers. The laptops that the team used could not be traced back to Uber, and each team member was given prepaid cell phones and WIFI devices for communications.

The SSG and MA have now raised the level of suspicion in the Waymo case as well as in the offices of the ATG. Sources have disclosed that both the MA and SSG were considered to be imperative since Uber's competition was also involved in industrial espionage and the overall feeling in the industry was of distrust. Jacobs letter and his court testimony claim that the data was stored in the detached devices to negate the surveilled groups from tracing back the information to Uber.

The evidence might support an extreme caution and even overzealous nature to how Uber managed their market intelligence gathering and security monitoring operations. However, in reality, Uber employees were targeted for violence in emerging markets and in some extreme cases, groups would discuss how to threaten Uber in events such as the one that Travis Kalanick was going to attend in India when he was still CEO. A group in Whatsapp was found out by SSG intending to attend the event and then set themselves on fire.

While SSG was intent on securing the safety of Uber personnel and drivers, MA was intent on scraping data from competitor websites, GitHub and Pastebin accounts and competitor API's and transferring the data to the offsite teams, that would sift and analyze the information that helped Uber understand how the competition worked, what their metrics were and give Uber an edge in the market. MA would also perform in-house "hacking" to test for vulnerabilities so that they could bolster their defense against competitors attempts at accessing their information. If they had done their job properly, then maybe the cracker who stile 57 million identity files would not have succeeded.

MA's work is what led Uber to incidences of confidential information theft, in one case outlined by Jacobs letter, leading to the theft of property code. Uber countered this claim by stating that the code was acquired through the public posting portal GitHub. GitHub is a portal that developers use to discuss code, and scraping the portal for code that is posted there is not a crime, in fact, that is the very essence of what is termed "free source code."

Eyeballing Hell

Eyeballing is a technique used to gather information from websites and apps and collect unique ID data of competing drivers and employees. The case of Hell for Lyft is a stand out example. Once the data is collected, it is merged with other data collected from other sources and the new larger file is then analyzed for a complete picture. The extra data sources would be bought from data providers, which is a nice way of saying "crackers" similar to the one that stole Uber's 57 million files.

According to Jacobs when an app is used overseas it would immediately transmit the driver's relevant data including their name, vehicle number, license plate number, phone number and email address. This information was then sucked into the MA system which would start to analyze the data and create a stream of information that showed how many rides a competitor was making, what incentives they were using, how much money they spent on rideshare driving initiatives and what was their actual market share. All the information gathered was sanctioned by Ubers legal staff and their CEO, Travis Kalanick. In fact, Kalanick stands accused of stealing information from Waymo when he dealt directly with OttoMotto's founder Alexander Levandowsky, who was a Waymo employee before he left the company stealing millions of files of information which he used to set up his AV venture OttoMotto, that was in a merger negotiation with Uber at the time Uber was partnered with Waymo.

The New Chief's

Uber's new CEO, Dara Khosrowshahi, and Uber's new Chief Legal Officer Tony West recently released an internal announcement, which Khosrowshahi sent to all Uber personnel;

Team -

I wanted to share Tony's note with the security team because I couldn't agree more with what he has outlined here.

The last 10 days have reminded us that things happened in the past that never should have occurred. The news that we failed to disclose a significant data breach, and that we showed poor judgment in our approach to competitors and our use of ephemeral communication for business purposes, has hurt the company just when we are beginning to turn the page.

I've already said that the decision not to disclose the breach was wrong, and we have held those responsible accountable.

With regard to the allegations outlined in Ric Jacobs' letter, I can tell you that we have not been able to substantiate every one of his claims, including any related to Waymo. But I will also say that there is more than enough there to merit serious concern.

As I hope you've seen over the past 2.5 months, I will always be fair when people admit mistakes or bring hard problems to me. But let me be clear: I have drawn a line. I will not tolerate misconduct or misbehavior that was endorsed or excused in the past. Period.

I want to close by saying that I couldn't be happier to have partners in Tony and the ELT as we work to build a company that every one of us can be proud of.

Onward … and Upward.


---------- Forwarded message ----------

From: Tony West

Date: Wed, Nov 29, 2017, at 11:42 AM

Subject: To My Security Colleagues,

As you all know, it's been less than a week since I started on this job. It's been an eventful period, to say the least, and I'm learning new things about Uber every day. One of the things that is constantly being reaffirmed to me is just how incredibly talented, smart, dedicated and loyal the people who work for this company are. I'm proud to be part of your team.

At the same time, in the spirit of transparency that I spoke about at the all-hands meeting yesterday, I must also say that I'm learning about practices we followed here in the past that are simply unacceptable; things that we cannot and will not tolerate at this company going forward.

A prime example is the failure to disclose last year's data breach to appropriate parties in a timely manner. Another I've just learned about in the last couple of days involves Uber security personnel engaging in the human surveillance of individuals who work for competitors.

Dara and I are still learning the details about the extent of these operations and who was involved in directing them, but suffice it to say there is no place for such practices or that kind of behavior at Uber. We don't need to be following folks around in order to gain some competitive advantage. We're better than that. We will compete and we will win because our technology is better, our ideas are better, and our people are better. Period.

My understanding is that this behavior no longer occurs at Uber; that this truly is a remnant of the past. And I have not learned anything in the last couple of days that suggests otherwise. But, to be crystal clear, to the extent anyone is working on any kind of competitive intelligence project that involves the surveillance of individuals, stop it now.

Let me also add that I've not learned of anything regarding the surveillance practices that would be considered illegal. However, as you will hear me say many times, the question for us is not just whether something is legal; we must also ask ourselves whether it's the right thing to do.

The data breach and human surveillance are the two biggest issues I've learned about in my short time here. If you are aware of or concerned about any other practices that could be questionable that occurred in the past or are occurring now, I expect you to raise those through the Hotline process immediately. We need to turn the page and begin writing our next Uber chapter, and I need your help to do that.

I'll continue to communicate with you about this and other topics in the coming days and weeks. As Dara said the all-hands meeting yesterday, we can expect some bumpy days ahead as more information about this and other past practices comes to light. But as much as Uber will be judged on what we've done in the past, people will be watching just as closely to see how we handle our response to these matters going forward.

And I'm confident that, working together, we will pass that test.



This message is aimed at individual surveillance, and demands that the old practices stop. However, Jacobs letter infers to digital mass surveillance. One interesting fact that has arisen and is public knowledge is that secure and detached servers are common place for cybersecurity when dealing with sensitive data. Uber did not do anything unusual in this instance. In fact, the National Institute of Standards and Technology recommends compartmentalizing sensitive data for all federal computer systems.

We add to this fact another fact; scraping is not an unusual activity, all companies in every sector "scrape" information to gain a better insight into their competition. Uber had blocked many instances when scrapers managed to access their systems, Uber considered it an affront and abusive but never perused the scrapers.

In a recent court finding, where Linkedin accused HiQ of scraping their member profiles as an action violating the Computer Fraud and Abuse Act. The federal judge presiding countered that HiQ did not violate the anti-hacking law. Scraping is the use of automated scripts that access websites public information and collects them into a database. This is not hacking as it does not enter a restricted area and does not violate the website's terms of use since the public data is "public."

Secret Messaging

While scraping might be legal, the use of encrypted and transient messaging is considered to be a precursor to illegal or unethical activity. If you have nothing to hide why would you use such a service. On the other hand, when dealing with delicate competitive issues, such as negotiations, some would consider using a service like Wickr expedient during the negotiation phase or when discussing secret issues when not being able to discuss them in the same location.

Wickr was hit by the blow back from the Waymo case and is not configuring in a script that will identify possible illegal messages and retain that information for law enforcement agencies. This will definitely reduce Wickr's market share since a secret messaging service that keeps information is no longer a secret service.

No matter what Uber does now if it uses a service like Wickr in the future, it will be considered a subversive act for some illegal and unethical purpose.

Uber's Hell and Jacob

Hell is the program that is damaging Uber the most. The federal investigation team heading the "hell" program lead has stopped MA from working and closed Uber's scraping system. While the MA and scraping systems are offline, their actions are now coming back to haunt Uber in ways that might be crippling.

Jacobs Letter includes a number of documents, the first one being a resignation message send by e-mail on April 14th, 2017 to then CEO Travis Kalanick, Salle Yoo, Uber's leading lawyer, HR head Liane Hornsey and Uber's new PR head Jill Hazelbaker. The message was titled "Criminal and Unethical Activities in Security." What had occurred was Jacobs then 37 years old, worked for just over a year in Uber's global intelligence unit was caught forwarding company e-mails to his personal e-mail account. Jacobs claimed he was building a case to expose Uber's activities.

Salle Yoo

However, the e-mail was then paired with a 37-page letter that Jacobs lawyer sent to Uber three weeks after the e-mail resignation message and it was a demand notice from Jacobs which came at a time when Uber sought anonymity from problems. Uber paid Jacobs $7.5 million to keep quiet; the letter reached the ATG in any event and from there ended up in Judge Alsup's hands.

The mainstay for Waymo was providing proof of their stolen data being stored on Uber servers. The introduction of Jacobs letter has excited Waymo's lawyers who are now zealously challenging the "secret" server system that MA used as being the place where their data was stored. Waymo has had no success in finding any evidence of information stored in Uber. Judge Alsop stated that the implications presented by the new letter showed that MA might be key in the case. The fact that Uber never entered the letter into evidence, or that they even mentioned their use of offsite servers made him furious, and he stated: "You stood up so many times and said, Judge, we searched our servers; these documents never hit a Uber server. You never told me that there was a surreptitious, parallel, nonpublic system that relied upon messages that evaporated after six seconds or after six days. You never mentioned any of that stuff. You never mentioned that there were these offline company-sponsored laptops."

Jacobs letters provide accusations that accuse MA of stealing Waymo data and storing it in their secret server system. In fact, the letter states clearly "Jacobs is aware that Uber used the MA team to steal trade secrets at least from Waymo in the U.S.,"

The Buy Off

When Jacobs testified in court, he said that he and his lawyer made a mistake and that MA never focused their operations on Waymo. Jacobs was asked by to explain why Halunen would insert such a statement, and Jacobs stated that he only skimmed through the letter while on vacation and that he did not stand by the statement that Uber stole information from Waymo. Jacobs added that "I don't think I did as thorough a job as I wish I could have"

This sudden reverse direction is understandable when reviewing the evidence that portrays Jacobs being bought off we are faced with the following:

The August negotiations with Halunen and Jacobs provided Jacobs with a $4.5 million settlement that was split into a $2 million cash payment and $1.5 million in Uber stock. Jacobs also agreed to be a freelance consultant for WilmerHale, a law firm chosen to spearhead the internal investigation into Uber's MA, to which Jacobs would receive a further $1 million. Uber also paid Jacobs lawyer a $3 million fee.


Why is Jacobs testimony so important to Waymo? The first reason is the fact that it was kept from the court. The second reason is who Jacobs was in Uber.

Judge William Alsup

Jacobs started working as manager of Ubers global intelligence team in March 2016. He directly reported to Uber's head of global threat operations, Mat Henley. Jacobs primary role was to review and parse data coming in from various sources in areas that Uber intended to open up operations and to ascertain threats to Uber and its operatives in that location. Henley was in charge of the MA team, although Jacobs had no access to them and in fact, his understanding of what and who they were only came from office chatter and word of mouth. Jacobs proposed that he gain access to the MA data frame to enable him to reach better conclusions. However, his requests were constantly resisted. In February 2017, Jacobs was reviewed by Henley, what was to be a standard employee performance review ended up in Jacobs being demoted.

Jacobs lawyer Halunen included this review in his letter "Jacobs experienced this review and demotion as pure retaliation for his refusal to buy into the threat ops culture of achieving business goals through illegal conduct, even though equally aggressive legal means were available to achieve the same end," However, according to Jacobs, his accusations were only based on first-hand discussions he had with Henley and the assumptions and conclusions he personally drew from those discussions.

The scorpion that stung itself

Angela Padilla, Uber's deputy general counsel, was the person that leaked the letter to the ATG as well as to the US Attorney's offices in the Northern District of California and Southern District of New York offices in September. Padilla stated in court that "We felt that Halunen was trying to extort the company, and I wanted to take the air out of his extortionist balloon. This case, given the huge sums of money that Mr. Jacobs was demanding at the outset, I felt was clearly extortionist, especially given the low value of his claims."

Angela Padilla

The action was taken by Padilla basically backfired, Uber claim the letter is a fabrication of the truth to extort the company, while Waymo took the latter very literally and seriously, as well as did the Judge. The fact that the ATG thought the letter pertinent enough to send to the Judge was enough for him to say "I know it's scandalous, but it's something that the United States Attorney thinks at least is true enough to give to me. If even half of what's in that letter is true, it would be a huge injustice to force Waymo to go to trial and not be able to prove the things that are said in that letter." Padilla's actions and the fact that Uber paid Jacobs so much money is pointing to proof that some of the letters if not all of it are true.