I have been observing Google being in some pain - search results are no longer visible which used to be visible, and sometimes "used to be" is "just a few hours ago".
I do not have enough insight to know how to fix these issues. I imagine there's a lot that needs to be done in a lot of different areas.
I am not even sure if all of these issues need to be fixed *within* Google. If Google is reflecting internet outages elsewhere, then those problems would need to be fixed elsewhere. (For example, my particular searches were focused on Debian - if Debian is failing for lack of interest on the part of its developers then maybe I ought to become involved with them again.)
Meanwhile, another issues has to do with the nature of management and corporate relations.
<personal introspective aside>
First, note that no one should want me to be a manager (unless someone who falls asleep in meetings and hates managing people is your ideal of a manager). I can help people, I can give them suggestions, I can see all sorts of good things in how managers deal with problems, but that's not me. I do *not* envy the job of a manager. Managers need to deal with all sorts of failures (that part is fun) but they also have a need to be accepted as being correct. And that need to be right can make things really difficult for them in a way that I do not enjoy. People are tricky to deal with.
I have found that I rather like people being upset at me, but only when I can then address the issues they are upset at me about. But I get stressed when someone gets upset at a person other than me (perhaps because those problems tend to be "out of bounds" for me to fix?).
Let's imagine that the problem included dead weight personnel within Google. How should that be dealt with?
One issue is that people tend to like to fix problems. If they are being dead weight quite likely they do not understand what problems they need to fix and/or how to fix those problems. Sometimes these issues can be fixed with some simple announcements or mentoring. Like, maybe:
"Google is doing great on 'hot topic' searches, but we are losing ground on 'long tail' searches. While it is completely appropriate to focus on fast responsive searches, we need to also have some way of returning results for more refined searches."
Hypothetically speaking, google should have two kinds of search backends - one which is constantly being updated, and another which is more archival. The archival backend would focus on "cool" content (see also "cool url") and would also be useful for isolating problems: recognizing spam, recognizing trends, and generally filling in the gaps in the "hot" content.
But let's say there are good reasons why Google doesn't currently do this. Regulatory reasons? Malicious hackers? Management priorities? Budget issues? Catastrophic bugs? Those reasons would be worth studying and investigating. For a persistent problem like this, there's going to be a trail of illogical decisions and subtle clues to follow and comprehend.
And, what to do about it?
Well, sooner or later, Google is probably going to have to split into multiple companies. It already has regional locations, so the trends are already in place. I'm not sure, though, how it would split. Would it get hit by a regulatory hammer? (Following the model set by Standard Oil and AT&T?) Or would it follow any of a variety of other models for how corporations spawn? (subsidiaries, franchises, contracting, etc. etc.) Honestly, I do not know.
<personal introspective again>
Personally, I have applied for some testing jobs - they seem ideal for me to learn about the problems from the inside - and I've also made a few noises about hiring on for Google X (since that potentially satisfies a "social good" urge I have nagging at me). My sense of ethics (or at least my lack of tolerance for conflicted goals - I hate disappointing people) has me only applying for work at Google, but that can't last indefinitely. Eventually, I will either hire on or start looking elsewhere and I plan on being very upfront about all of this with whoever I work with. If I take a job a company that does not want me to work for Google, that will be that until they are done with me. If I take a job with a company that is comfortable with me heading off to Google, I think I would be ok with that (but whether I switch jobs would depend more on how much fun I think the job would be than the payment arrangements).
My last Gannett paycheck was yesterday, and I will need to file taxes before I have a really good idea of what my finances and budget will look like and how that will effect my job possibilities. I am having fun working for nothing, I just need to watch out so that I do not procrastinate too long.
* * * * * old version follows * * * * *
Google, it seems to me, is being raped, by success. (Or at least that is how I, as a rape survivor, view the current situation.)
And this matters. It matters for me (or might in some number of months, when I need to find a new job) because information resources that should be available are not. It matters for you because you also need some kind of access to information.
But going from an observation to a useful plan of action can be tricky. If you solve a problem the wrong way you get the wrong kinds of failures. From my point of view, failing to fail is a good kind of failure, and failing at finding publicly available information which was available a few years ago, or an hour ago [and I've been experiencing both, a lot] is the wrong kind of failure for a search engine provider like Google.
So, what to do?
Well, if success induces failure one clear solution is more success, eventually the failures will come back and get rid of the problematic success. Or, in plain english: this current strategy is going to eat Google's bottom line (and I have also seen what I consider significant evidence that it already has been eating Google's bottom line). But keep in mind that other people *will* have different perspectives from mine.
See, one issue is that one individual can only do so much. But another issue is that different people will have different ideas of "good" and "best". So Google's sheer size is going to result in different groups of people pushing in different directions.
Put differently, thinking of Google as an organism, it's maybe approaching time when it's time to split apart, or give birth or something of that character. Of course, in some sense it already has been - with the various regional locations. But the more successful it is the more regulatory pressure it will experience and just like Standard Oil and AT&T we can imagine something similar in the future for Google itself.
Another approach might involve reaching out and creating voluntary working arrangements with other search engines. Of course, this gets back to potentially looming regulatory issues.
But those regulatory issues are not the only problem. A real issue has to do with the flavor of information and the nature of exchange systems (such as money) and the nature of success.
Once upon a time, the internet was largely academic in nature - the information available thus intrinsically had an academic slant and this reflected in search results. And then came spam (high volume, low quality advertising), malware, social media (low volume, varying quality interactions), dating sites, etc. etc. And lots of money. How to manage this all?
I have some ideas, but a lot of them are still pretty diffuse, I might not be the right person to really solve these problems. But I would (rarely) like to have the option of searching for spam, (more often) like to have the option of searching for advertising, (quite often) like to have the option of searching within the prepackaged search classification system itself to find the right way of expressing to the search engine what it is I am currently trying to search for.
But none of that is really meaningful if the content behind the searches is offline. So maybe more funding is needed for historically valuable sites (the wayback machine, for example, and/or also long running sites whose robots.txt do not exclude search engines and whose content seems to be stable over time). Maybe instead of (or in addition to) hiring people and buying up promsing companies, Google should be giving prizes to the most valuable website of the year, or whatever else.
And, in the long run, Google is going to need to address other forms of information and other modes of information delivery and other concepts of searching for information. But it scares me that information that I used to be able to find easily, through Google, is now offline. (And I have tried other search engines also - they are not necessarily much better.)
In the shorter run, it might be good if we see more hints about how Google has classified our search. And I would really like to see some kind of "historical search" option. Maybe this is slow and maybe this should not be the default - people like quick answers - but when I try to refine a search (one that used to get a million hits) and I get nothing? Something is very wrong.
But solving these strategic issues does not really interest me, personally. Or not much. These are large scale problems and will require a lot of people acting together to resolve them. And I am just one person. So while I can be aware of these things and give my perspective on them, they only attract my attention as a passing interest.
Also, I currently only have an outsider's perspective on Google's issues. What I'd really like would be to get in and admit total ignorance of how things work. I'd like to work on the systems Google uses internally to find failures in the search engine itself. And by failures I do not mean these administrative failures or strategic failures but actual machine failures. I learn by doing, and I have a lot to learn.
But maybe not this month. I have a variety of other things which also are demanding my attention. Problems which need solving. And I expect that it is best, for now, if I deal with some of those.
Still, this kind of thing keeps nagging at my awareness, so I have been thinking long and hard about why I am seeing this. Why is this happening? Hackers? Neglect? "Privacy"? Regulation? The sites being referred to going down?
I imagine it's a bit of all of those. Or maybe Google is just too successful to sustain itself?