Life With Larger and Larger Files

From the days of floppy discs holding kilobytes to Zip discs holding megabytes to thumb drives holding gigabytes, we appear to have come a long way in storage technology — so much so, that we were once excited about our data being “big.”

Yet, try to transfer large files over the Internet, and obstacles, headaches, and costs begin to appear out of nowhere. Our ability to generate content seems to always be a step ahead of the infrastructure’s ability to store it.

The problems become most apparent when you have to work on large files, putting pressure on local storage devices while also revealing the constraints of file transfer speeds.

Storage costs have dropped precipitously since the 1950s, when a terabyte of disk storage cost billions of dollars. Now, the same storage is about $14. Interestingly, we may be reaching a threshold, as costs for memory, solid state storage, and disk storage appear to have flattened out:

Another source estimates that from 2017 to November 2022, the average cost per gigabyte storage on hard drives decreased by 56.36% (from $0.033 down to $0.0144).

Despite this, we’re spending more than ever on storage media, a trend that is projected to continue through 2027, based on this projection of the revenues of memory manufacturers:

Part of the move to USB-C as a standard for powering and connecting smartphones is that it allows storage device dongles to connect to your phone, another sign that the storage challenges are expected to be a persistent problem.

The pressure is coming from the demand side — with billions of devices shooting video of vacations, concerts, birthdays, and weddings, and now with scads of virtual meetings, webinars, and conferences being recorded, millions of large files are being uploaded throughout any day around the world.

Producing content is our new result, job, obligation, habit, or hobby (depending on how you look at it), and the volume of data flowing across the fiber and into fixed media is growing because we’re also making bigger files. As the iPhone 15 has shown us, our pursuit of sharper images also leads to more charges for iCloud and other solutions as the photo files gobble us scarce local memory resources.

The headaches for large file handling are numerous. You can run out of storage in the Cloud or on your device. Slow upload speeds can cause timeouts. Servers may not be configured to support the files. Email clients have variable limits on the size of attachments. Solutions like iCloud can bog down your workspeed as files are moved into and out of the computers Apple owns and operates.

In addition to pushing things to social or sharing files for work, we often produce and share casually within smaller social circles, and new solutions like cached storage for texts containing large video or image files, downloading only when actively requested, only underscore the scarcity problems with bandwidth and memory.

Dropbox and similar filesharing solutions are difficult to use — logins proliferate quickly, plans get wonky, and cost tiers add up.

Google Drive has a trick in it to drive revenues — the person sending files has to have the capacity to send the size of files, in addition to the person receiving them.

Hard media — thumb drives, SD cards, and similar — all require people to meet physically or use some sort of delivery service.

Part of the problem is the pure physics of connection, with upload protocols far slower than download because the infrastructure privileges downloads.

Coming from an era in which people boasted about the limitless capacity of the Internet and its amazing speeds, it’s fascinating to experience again and again how difficult it can be to upload, store, and share large files, and how we are perhaps reaching some harder limits on capacity.

The Internet right now seems to be running up against physics, economics, and social priorities in ways that we should think about carefully in order to keep the good parts of it working well.

• Note: Speaking of the pressures to produce content, I came across this nugget related to the music business. It seems to apply more generally, including to scholarly and scientific publishing, which has become a volume-based exercise more than I ever imagined it would. As someone who has produced a lot of content for the past 15 years, it makes me grateful I have pulled back from the endless and counterproductive drudgery of social media, and can write only when I feel I have something to say.