Preserving The Internet: The Wayback Machine

The Wayback Machine

News broke this week that The Wayback Machine has now archived 4 hundred billion webpages. The Wayback Machine enables users to see how another website looked in the past, and after launching in 1996 the project now requires an incredible 5 petabytes of storage to maintain all its data.

The figures behind the project are impressive; The Wayback Machine’s database is queried over 1,000 times every second by over 500,000 people a day – making Archive.org the 250th most popular site on the entire internet.

The company’s mission is equally impressive, explicitly expressing their ideology on their website “Most societies place importance on preserving artifacts of their culture and heritage. Without such artifacts, civilization has no memory and no mechanism to learn from its successes and failures. Our culture now produces more and more artifacts in digital form. The Archive’s mission is to help preserve those artifacts and create an Internet library for researchers, historians, and scholars”.

Brewster Kahle launched the project in 1996 at the same time he started the now-famous web crawling company Alexa Internet. The project has its roots in the development of software that could crawl and download all publicly accessible World Wide Web pages, the Gopher hierarchy, the Netnews bulletin board system, and downloadable software. The archived content itself wasn’t available until 2001 but by 1999 the archive had already expanded its collections to include texts, audio, moving images and software.

Uses of such an archive are widespread. There is a day-today practical use, as evidenced when The Wayback Machine provided access to important Federal Government sites that went dark during the Federal Government shutdown in the United States. There is also an educational aspect, with importance lessons to be learned from the vast amount of big data stored within its archives. Finally, there is a historical aspect, as the development of our internet has been preserved for future generations to enjoy.

The cloud-based project has not been without its controversies. In 2012 China restored access to the database after blocking it for several years, while in the USA an activist sued the organisation for $100,000 after claiming that archiving her site breached her terms of service. The dispute was ultimately settled out of court.

The success of the project has started to attract imitators. Online companies such as Archive.It, Freezepage, and iTools all offer similar services, but not of them can offer the same quality and depth of content as The Wayback Machine.

Is this a vital project or a waste of valuable storage space? Are there ethical questions surrounding the unhibited archiving of so many sites, or are there taking a virtual photograph of events? Let us know in the comments below.

By Daniel Price

Juan Pablo Perez Etchegoyen

The S/4 HANA Decade is Here: Three Tips for a Successful Migration

Three Migration Tips For organizations using SAP, migrating to S/4 HANA is a project that’s either in the works or on the horizon as the 2027 deadline for completion looms. The new generation of SAP ...
Gary Taylor

5 Reasons Why Virtual Desktop Infrastructure Will Go Mainstream Post 2020

Virtual Desktop Infrastructure Growth Virtual Desktop Infrastructure (VDI) technology enables remote users to access their desktop from anywhere using an internet connection. This technology has been around for a couple of decades but never received ...
Ronald van Loon

Cognitive Science + Data Science: Next-Gen Data Driven Marketing

Next-Gen Data Driven Marketing What are the psychological motivators driving customer purchasing decisions? The era of ubiquitous digitalization, advanced analytics, and big data is here. Yet marketers frequently grapple with obtaining information that answers critical ...
Kaylamatthews

How AI Can Keep Documents Secure During the Age of Remote Work

Keeping Documents Secure While remote In response to the COVID-19 pandemic, global businesses have restructured operations to accommodate remote work and telecommuting. Humanity has collectively separated and isolated to prevent the virus's spread. In under ...
Sangeeta Chhabra

Leverage DaaS To Solve Challenges of Remote Work

Solve Challenges of Remote Work In the past one year of Coronavirus (COVID-19), we have seen it all: An ailing world economy, dramatic changes in the political structure of many countries, and, most prominently, a ...
Kokumai

Identity Assurance – Sufficient and Necessary Conditions

Identity Assurance It is not easy to define the 'sufficient condition' for describing a set of processes used to establish that a natural person is real, unique, and identifiable; criminals keep coming up with hitherto ...