Inside the curious world of physical data transfers

Walter Scott laid a hand on the white semi-truck parked outside DigitalGlobe’s headquarters in Westminster, Colorado.

It was 2016, and DigitalGlobe—a commercial satellite-imagery leader with upwards of $700 million in revenue—was the first-ever customer for Amazon Web Services’ “Snowmobile,” a new way to physically transfer large amounts of data: via a 45-foot shipping container on wheels.

The truck held not only Scott’s life’s work, but also that of the company he founded: a satellite image library with 16 years of Earth and space records.

It had taken a couple of months to get everything ready and then upload the company’s entire satellite image library—10 petabytes of data, or 5x the amount contained in all US academic research libraries—to the Snowmobile. After the data was translated and encrypted, the truck headed back to a cluster of Amazon data centers to upload it to cloud storage.

Scott gave the 18-wheeler one last pat before it drove away.

For DigitalGlobe, this was a high-stakes bet and a one-time occurrence. For AWS’s data-transfer truck, it was the inaugural journey of a high-dollar data-transfer service. And while other tech giants may lack the 18-wheeler and armed guards, they also tend to offer a range of physical data-transfer options for customers with lots of information to migrate, limited connectivity, or both—e.g., data that could take months or years to upload online.

“Maybe you have hundreds of petabytes that need to be moved, but it’s a long-term process—you’re moving a data center,” Wayne Duso, AWS’s VP of storage, edge, and data governance, told us.

The global cloud computing market is a $270 billion example of how physical infrastructure—servers, data centers, and, yes, data-transfer devices—powers not only the tech we use every day, but also the data storage that allows those platforms to operate.

Let’s get physical

Here’s how the process starts: The data-transfer device is shipped—or in Snowmobile’s case, driven—to the customer, who loads it up with data before securely sending it to the company’s upload facility. After the data is processed, the device is wiped before being sent out on its next job. But the process doesn’t come cheap.

Depending on the cloud-storage provider, the customer, the geographic region, and the amount of data to migrate over, the process can be pricey—for example, Google’s 40-terabyte device costs at least $300 for 10 days of use (plus shipping). Physical device options range from shippable, handheld hardware to AWS’s hulking 18-wheeler.

Think of it like moving day—but instead of boxing up your clothes and books, you’re packaging up data for cloud storage. And whether you need a suitcase or a moving truck simply depends on how much stuff you’ve got.

Jay Littlepage, then a VP at DigitalGlobe, told us that before the company struck a deal with AWS, its approach to customer data transfers wasn’t scalable. If someone wanted HD imagery of Paris, for instance, DigitalGlobe had to send it via FTP or by shipping a firewire drive. “We ended up actually turning down a lot of business because somebody would want something by a certain date, and...the laws of physics said we couldn’t do it,” Littlepage recalled.

Google’s Transfer Appliance, a server-sized device and the company’s only option for physical data transfer, can typically be used for between 100 and 480 terabytes. Nuro, an autonomous delivery and robotics company, uses Transfer Appliance in its garages. The devices store the robots’ sensor data, and Nuro can drop-ship the devices back to Google in order to upload the data to the cloud and help train its algorithms to deal with different real-world scenarios.

“Most businesses have some type of network connection, but they use that network connection to run their business,” Brian Schwarz, director of product management for Google Cloud Storage, told us. “You don’t want to clog that pipe. Think about this network connection like a highway...If you, as a data transfer, have some job that’s going to create the epic traffic disaster or traffic jam of all time, and all your employees are basically going to be locked out of using the network for a week, that’s not really tractable.”

Amazon’s smaller offerings—the briefcase-sized Snowball and Snowball Edge—can hold up to 100 terabytes of data. Meanwhile, Snowmobile has the capacity to move up to 100 petabytes of data in one trip. AWS declined to provide the pricing range for the service.

Securing the valuables

Since physical data transfers typically involve massive amounts of sensitive information—historical datasets, trade secrets, company records, and more—security is paramount. So the devices are typically fortified against both cyberattacks and physical tampering. Google’s Transfer Appliance, as well as AWS’s offerings, uses 256-bit encryption.

Snowmobile, for its part, is both waterproof and fireproof. It offers an unmarked escort vehicle, crewed with inconspicuous but armed security guards who work in shifts. The data is safeguarded with customer-provided encryption keys, which erase if the vehicle loses power.

That could have happened on DigitalGlobe’s Snowmobile journey, Littlepage recalled, when a snowstorm threw the vehicle for a loop: AWS had rented a chiller to keep the devices inside cool, but the chiller turned off upon sensing the cold temperatures outside the truck. The interior heated up, and AWS had to send out a technician to replace some hardware before the Snowmobile could continue its road trip to the data centers.

As the amount of data created every day continues to increase—and retail, manufacturing, and other industries collect and analyze more high-definition video than ever—companies will have more and more sensitive data to store, meaning an uptick in data storage and the environmental burden that accompanies it. We’ll likely see physical data-transfer options expand as well, from cloud storage giants expanding their product line to smaller companies scaling their first foray into the space.

“The need for much bigger data-set sizes...has been a trend for the last five or 10 years—it’s absolutely going to progress in the future,” Schwarz told us. “We have healthy R&D, [and] the most recent appliances we introduced are absolutely not the last ones. There will be bigger, faster, better ones coming down the pipe in the future.”