Then again, you can get a gig of memory on a keychain these days. I’m so old I remember the first one gig drive, it was the size of a washing machine. And we were so excited to see it! And before that an entire roomful of computers couldn’t give you a gig of memory. But I also remember the first transistors my dad brought home from Motorola, and then getting a little transistor radio and what a wonder it was. That probably really started my love of engineering.
As I finally get my eldest off to a real university to study computer science, I wonder what his generation will create. Looking forward to more good stuff, but honestly, this Internet is what I’m proudest of my generation doing. Back when I was on Multics forum, this was just the way we techies communicated. Now everyone can use it. I’ve sometimes regretted that, especially the spam, but when we can use it in the right ways, it is indeed a wonder.
We’ve known it for a long time: the web is big. The first Google index in 1998 already had 26 million pages, and by 2000 the Google index reached the one billion mark. Over the last eight years, we’ve seen a lot of big numbers about how much content is really out there. Recently, even our search engineers stopped in awe about just how big the web is these days — when our systems that process links on the web to find new content hit a milestone: 1 trillion (as in 1,000,000,000,000) unique URLs on the web at once!
How do we find all those pages? We start at a set of well-connected initial pages and follow each of their links to new pages. Then we follow the links on those new pages to even more pages and so on, until we have a huge list of links. In fact, we found even more than 1 trillion individual links, but not all of them lead to unique web pages. Many pages have multiple URLs with exactly the same content or URLs that are auto-generated copies of each other. Even after removing those exact duplicates, we saw a trillion unique URLs, and the number of individual web pages out there is growing by several billion pages per day.
So how many unique pages does the web really contain? We don’t know; we don’t have time to look at them all! 🙂 Strictly speaking, the number of pages out there is infinite — for example, web calendars may have a “next day” link, and we could follow that link forever, each time finding a “new” page. We’re not doing that, obviously, since there would be little benefit to you. But this example shows that the size of the web really depends on your definition of what’s a useful page, and there is no exact answer….