- 7 Degrees of Freedom
- Posts
- How Tesla used robotics to survive "Production Hell" and became the world's most advanced car manufacturer
How Tesla used robotics to survive "Production Hell" and became the world's most advanced car manufacturer
Jack Pearson
August 12, 2022
Welcome to The Robot Remix - giving you strategic insight into the robotics and automation industry with weekly news bulletins and monthly deep dives by Remix Robotics.
Introduction
In 2018 Tesla was on the brink of bankruptcy. They had invested billions into a new factory filled with robots that hardly worked and couldn’t meet production quotas. Today Tesla is the 6th most valuable company in the world and its heavily automated production process is ten years ahead of competitors. How did they build themselves back from the brink and turn robotics from strategic failure to success?
This article will review how Tesla’s automation strategy has shifted over the last five years. By investigating where Tesla made mistakes and where it excelled, the reader will benefit from Tesla’s hard-earned lessons and gain an understanding of how to build an automation strategy.
In this article, we will cover -
The Context - How Tesla differs from other car manufacturers and their 2016 goals for the Model 3
Tesla’s 2016 Automation Strategy - How integral automation was to their production goals and why it failed
Tesla’s updated Automation Strategy - How Tesla improved their approach to automation and became best in class
The Context
A detailed introduction of Tesla and its famous/infamous founder Elon Musk shouldn't really be required. Tesla is one of the best known car manufacturers and basically kicked off the electric vehicle revolution.
Their success has been due as much to marketing and fundraising as their engineering innovation. Tesla's approach has always been to make impossible plans in public and successfully follow through. This strategy has led to great success and - as we'll see, has also taken them to the brink of bankruptcy.
The bold predictions started in 2006. When Tesla was still a start-up they published a “Top Secret” plan for making electric vehicles the industry standard -
Build a sports car
Use that money to build an affordable car
Use that money to build an even more affordable car
While doing the above, also provide zero-emission electric power generation options [1]
This memorable strategy highlights their marketing ability but underplays all of the challenges that an early-stage start-up would face developing a new car from scratch, let alone a new car built around a novel power system.
Making Cars is Harder than Rocket Science
Modern automotive production is complicated. Cars have thousands of sub-components sourced through complex global supply chains. Integrators must invest in expensive and specialist equipment requiring high production volumes to ensure affordability for consumers.
As the Founder of SpaceX, Musk is well placed to know cars may be more difficult to produce than rockets. Rockets are obviously an extreme challenge in their own right but they are made slowly, in low volumes and can be checked meticulously by hand. In comparison, automotive production lines spit out a new car every minute. These vehicles need to be faultless and capable of surviving decades in all weathers and environments. Even minor failures can result in catastrophic loss of life. As a result, automotive manufacturing is traditionally the domain of large, established companies.
Rather than seeing this as a deterrent, Musk saw a bloated industry filled with stale incumbents and even staler processes - all ripe for disruption.
An Industry Ripe for Disruption
Musk has become famous for using a First Principles approach to problem-solving. First-Principals thinking refers to the methodology of boiling down a problem into its core assumptions and questioning anything that doesn’t conform to fundamental physics [2]. The assumption that cars had to run on petrol and only established players were suited to automotive manufacturing was thrown out the window. Tesla even questioned the standard way of building vehicles with its complicated network of suppliers and integrators. Instead, they decided to follow an Apple-like strategy of vertical integration. This meant designing and producing as much of the car in-house, the exact opposite of the industry orthodoxy.
The advantages they hoped to gain from this approach included -
Improved customer experience - Controlling more of the process leads to greater control of the quality and the ability to ensure components are optimised to work together in a way that delights users
Increased profit margins - Eating away at the activities of suppliers, or ‘barnacles’ as Musk calls them, turns their profit margins into Tesla’s
Reduced bottlenecks - In-housing slow to source or often unavailable components like batteries allows Tesla to find solutions to supply chain issues. Tesla has taken this to the extreme by mentioning that lithium mining may one day be part of their remit.
This approach worked successfully for Step 1 of the secret plan. The earlier models - Roadster, Model S, and Model X are all luxury vehicles produced at lower volumes with the ability to use manual labour to ensure high quality was maintained. Sidestepping many of the challenges of automotive mass production.
In 2016 Tesla decided it was time to move to Step 2 and develop a more affordable, high volume car, bringing itself into the “real car company” club. They launched a wildly successful presale for its sedan style Model 3, generating 325,000 orders ($11.4 billion). To meet these orders, Tesla was promising to produce 5000 cars/week by 2018[3]. Putting this into perspective, this was a commitment to build a new and relatively untested car filled with innovative features at a rate of 1 car every 2 minutes. To complicate matters, Musk was also set on completely changing how these cars were being made - which is where automation comes in.
Tesla’s 2016 Automation Strategy
Tesla’s production strategy for the Model 3 can be broken up into two core facets -
1) The “Alien Dreadnaught”
“Our internal code name for the factory, the machine that builds the machine, is the alien dreadnought, when our factory looks like an alien dreadnought, then we know it’s probably right.”
To meet Tesla’s uncompromising production goals, Musk settled on an equally uncompromising production strategy - full automation, no ifs, ands or buts. The goal was to remove every single human operator and only stop when the facility looked like an alien spaceship. The logic was that increasing the volume and velocity of vehicle production required moving from “people speed to robot speed”. Using First Principles, Musk reasoned that if mechanisms can undertake tasks faster, more consistently and more accurately - then Tesla should aim for as much automation as possible and do so as quickly.
2) Build Hardware Like Software
Musk came up as a software entrepreneur and his approach to hardware is heavily influenced by agile software development. Silicon Valley essentially runs on the Lean Startup approach, rather than building something alone in a dark basement, the goal is to get a product into customers’ hands as quickly as possible to understand their needs through tangible experience and iterate your product to meet these needs. This is ideal for software as it’s quick and easy to update a product and redistribute it without costs or friction. Reid Hoffman, the founder of LinkedIn & Musk’s colleague at PayPal, epitomised this approach by saying -
“If you are not embarrassed by the first version of your product, you’ve launched too late”.
For Tesla, following this approach meant shortening development cycles, forgoing trial production runs and diving straight into full-scale production. That is, full-scale production of a cutting edge product at the edge of feasibility, at volumes 100X higher than they had experienced, using a new and untested production approach. Unfortunately, they don’t call it hardware for nothing... the same rules do not apply.
The Result - ‘Production Hell’
In late 2016 Musk set about converting an old GM / Toyota factory in Fremont, California, into his Alien Dreadnaught. Tesla gave the Model 3 engineers free rein to redesign the site for mass production. They invested in developing in-house robotics capabilities by acquiring two automation companies, Grohmann & Perbix.
With the vision of creating an alien spaceship firmly set, Tesla set about automating everything in sight. They purchased over 1,000 robots including 6-axis arms from Kuka and Fanuc and automated vehicles from Omron. They applied them across the process tackling relatively routine tasks such as welding or painting through to entirely novel tasks like wire harness assembly.
To illustrate how ambitious Tesla's goals were let's look at wire harness production. Wire harnesses are essentially bundles of individual wires that transmit power and data across the vehicle and there is a reason they're almost exclusively made by hand. As you can imagine, manipulating a wire is challenging for a mechanical system - they're small, thin and flexible. This is made even more challenging when you consider the wires need to be cut to different lengths, stripped, crimped and joined by various types of connectors. To make matters worse a single harness can be made from 1,000s of wires. Automating this process would be a feat of engineering if it was a company's sole focus.
By April 2018, everything was not going to plan. The company was far behind schedule, only producing an average of 2000 cars/week and haemorrhaging $100 million dollars a week. Amid mass resignations, health and safety incidents and general public ridicule, Musk dubbed the period “Production Hell”.
The robots were not panning out as hoped. They struggled to achieve the required throughputs and weren't hitting quality requirements. Small inefficiencies compounded and resulted in substantial delays. As a stop-gap, Tesla hired 100s of temporary workers to pick up the slack.
Why Did This Happen?
Much of Tesla’s production hell was blamed on excessive automation but what caused such a significant deviation between expectations and reality? Why did their automation strategy fail?
Tesla tried to do everything all at once. They had a new product being made by a new process at a new site. At each level, Tesla was pushing the boundaries and going against orthodoxy.
New Site
The first constraint that Tesla faced was the site itself. They did not have a blank canvas in Fremont. The facility was initially opened in 1962 and required a complete overhaul to convert it into the hyper-automated electric vehicle production machine of Musk's dreams. Tesla had the budget for this redesign but due to its presale promises, it did not have the time.
New Product
The Model 3 was jam-packed with features that were new for Tesla and unique for the overall industry. They were doing this at a lower price than any of their other models.
The level of innovation resulted in updates and changes. Due to the tight deadlines, these occured during the production ramp-up. In such a complex process, tiny changes create minor variances, which over many steps can compound and have huge impacts on upstream operations.
A prominent example includes the battery cells which were updated from the Model S. The cells had improved capacity which resulted in a slight increase in size over the standard 18650 cells used in previous Models. An automated process had been designed to pack cells into their trays, 50% faster than human operators could[4]. Unfortunately, the change in cell size had not been accounted for and resulted in increased error rates, requiring manual operators to take over the system before it was redesigned.
New Process
In many ways, the factory design was as innovative as the product design. Automation was being relied on for everything from incredibly precise and dexterous tasks like wire harness assembly to heavy-duty tasks like lifting entire vehicles. The Fremont site replaced the original gantry systems with 10 of the largest robots available and in typical Tesla style, named them after X-men characters.
The innovation across both the product & the process was not given enough time and testing to ensure reliability and repeatability. The frequent changes & variability could not be accounted for and attempts to make the automation generalisable with machine vision and other intelligent technologies were unsuccessful.
As Bill Gates stated in his first & second rule of technology -
Rule 1: Automation applied to an efficient operation will magnify efficiencies
Rule 2: Automation applied to an inefficient operation will magnify the inefficiency
Meeting the appropriate quality levels required a finely tuned production system but -
Innovation and uncertainty across the production process,
Ridiculous production targets and an extreme ramp-up,
An unmanageable number of new and insufficiently tested robots
made it impossible to find enough hours to refine each robot cell and ensure it met its required speed and quality.
Tesla’s Updated Automation Strategy
During the Model 3’s production, Tesla’s future seemed in question. With bankruptcy a real possibility, automation appeared to be one of the major culprits. Today things are a little different - Tesla did not go out of business and successfully reached its production targets. More than just scraping through to survival, Tesla is now thriving - the Model 3 is regarded as one of the best cars ever made[6], Tesla became the 6th company to reach a trillion dollars and as of April 2022, Musk is the richest man on earth.
Patient customers and goodwill from deep-pocketed investors played a factor, but in the end, self-imposed constraints and challenges led to innovations in production that have paid off 100X. Sandy Munro, a manufacturing consultant (famously not a good news consultant) who had previously been critical of Tesla’s design, changed his option after completing a teardown of the new car -
“I said in the past that Tesla with the Model 3 was probably five to eight years ahead of everybody else,” he said. “And now, in some cases, I think that in some areas of the car, Tesla is 10 years ahead, especially when it comes to the manufacturing.” - Sandy Munro
Rather than lose faith in robotics after all of the friction and integration challenges, Tesla has subtly reshaped their approach while doubling down on automation.
Their new strategy can be broken into three core tenants -
The Factory is the Product
Automation in Sequence
Assess Opportunities Objectively
The Factory is the Product
Tesla realised that process design was much more challenging than the actual product design. Proof of concepts and low volume, luxury vehicles are forgiving, and problems can be ironed out leisurely by hand. Full-scale production is not as lenient -
Cost of errors increase throughout the design process and peak once expensive fixed tooling and automation has been integrated
Expensive fixed equipment requires high volumes to reach profit
High volumes require little to no down time - time is money and delays / rework cost lots of money
Complex, multi-stage operations create positive feedback loops and emergent properties - bottlenecks appear in one part of the process and propagate unpredictably upstream compounding the problems and grinding the whole plant to a halt
This realisation caused Tesla to change their focus - the car isn’t the only product. The factory is too. The factory requires as much design, testing and validation as the car, if not more.
“We’re bringing a massive amount of effort into manufacturing and engineering the machine that makes the machine. There’s probably 1,000%, maybe 10,000% more engineering required for the factory than for the product itself.” - Elon Musk
Automation in Sequence
During a 2021 SpaceX factory tour, Musk explained his new mental model for production derived from his experiences in Model 3 production hell[6]. He realised that in many ways, the challenges had come from incorrect sequencing - he had designed the production process in the wrong order. Now he tries to implement this 5 step process across all his hardware companies -
1. Make The Requirements Less Dumb
All designs are wrong. It’s just a matter of how wrong. Everything stems from requirements - unnecessary, unclear, and incorrect requirements will have significant downstream impacts. Nail these down before moving on or else.
2. Try And Delete Part Of The Process
Engineers tend to fall in love with technology and forget to question whether something should even exist. Scope creep and bloat lead to excess complexity, and unnecessary work and amplifies the impacts of errors. Ruthlessly cut out features and requirements.
3. Optimise
Only optimise a process once the bloat has been removed. Iron out the errors and quality issues to ensure consistency and stability.
4. Accelerate Cycle Time
Then and only then, speed things up. If the quality isn’t there, stop, don't speed things up. “If you’re digging your grave, don’t dig it faster”.
5. Automate
A process needs to be stable and validated to meet a company's goals before automation can be implemented - See Bill Gates's 2nd Rule of Technology.
“I have personally made the mistake of going backwards on all five steps multiple times. In making Tesla’s Model 3, I literally automated, accelerated, simplified and then deleted. It was like being in a Dilbert cartoon." - Elon Musk,
Assess Opportunities Objectively and Iterate
The last change in strategy was the realisation that automation is not a panacea. There are some benefits to “Lights out Production” but full automation should not be seen as an end in itself.
Instead, each aspect of the process should be assessed independently and objectively. A cost-benefit analysis should be undertaken to compare the different technological solutions with the human equivalent. The system that best meets production goals should be selected (see Remix Robotics Discovery Process).
Engineers can be drawn in by the latest gizmos and gadgets but need to acknowledge that, for the time being, humans are very good at certain tasks (high dexterity, high feedback, high variance). That said, the industry is evolving rapidly. Technologies that wouldn’t meet Tesla’s requirements in 2016 have advanced to the point where they are likely to, e.g generalisable machine vision.
When automation is best placed to solve a challenge, it is essential to assess the level of innovation or technology risk involved. The greater the innovation, the more testing and iteration required. Don’t expect to throw R&D technology straight into full production without challenges. At Remix Robotics, we recommend an iterative development process building from prototype to proof of concept to production system with testing and iteration at each stage.
Key Takeaways
It may have been an uphill battle but the results of Tesla’s new strategy speak for themselves. Gigafactory Shanghai is outfitted with 445 robots on every line and can reach peak manufacturing speeds of 45s / Model Y. This surpasses Ford’s famous production record of 53 seconds to produce an F-150 truck and represents a milestone for the industry. What can be learnt from Tesla’s automation journey -
Mass manufacture is 100X harder than low volume production. Use conservative estimates to ensure that adequate time and budget remain to refine the process successfully.
Push boundaries but minimise risk. Unless a company has Tesla’s coffers, innovating across all fronts creates a lot of risks. If a product is immature, be strategic about where you implement novel manufacturing processes.
Automating a flawed process makes it worse. Robotics is a force multiplier. If it is applied to a good process, it is improved and the inverse is also true. Make sure to implement automation at the correct stage of development.
Automation brings a clear competitive advantage. Robotics when applied correctly can bring costs down, increase quality and allow companies to meet demand.
“Based on every measurable metric, Model 3 is already the highest quality vehicle we have ever produced, and this is unquestionably due in large part to automation” - Elon Musk
Jack Pearson is the Commercial Director and Co-Founder of Remix Robotics - an automation design agency that builds custom robotic systems for companies including DHL, Mercedes F1 & the Small Robot Company.
Disclaimer: Note many factors drive success and failure. This article only references publicly available information. Nothing in this article constitutes advice...