(Photo Credit: Gizmodo)

You lose some. You win some: Of course as NBC’s online partner, Microsoft gets a least a cut of the $100 million dollars in online advertising spent around the Olympics. And the millions of downloads of Silverlight aren’t too shabby either.

The Internet is Falling! Arbor Networks, a security and network management company, partnered with ninety network services and content providers from around the world to publish an extensive study of IPv6 traffic on the Internet. Craig Labovitiz, Arbor Networks chief scientist, stated that only 900 days were left until the end of the Internet, or at least the exhaustion of IPv4 registry allocations. For the past year, the study shows very little IPv6 traffic – something like 1/100^th of 1% of Internet traffic. Craig credits this to money issues. “The department of commerce estimates it will cost $25 billion for ISPs to upgrade to native IPv6.”

Blogger James Urquhart created a bill of rights for cloud computing. The purpose of the bill is to “help guide would-be cloud customers to those clouds best able to guarantee their freedom.” The blogosphere is a great place to get some open debate going, and I applaud James for trying to make something yet so “cloudy” a bit more clear and concrete. But what’s up with the creating a PAC for this?? (Check out the comments.)

Trying to get by on limited resources? Need more money, staff and the freedom to focus on long-term projects? Sound familiar? Then you just might be in IT at a midsize company. (or in marketing at a young but rapidly growing IT company ) Arrow Enterprise Computing Solutions conducted a survey of 200 tech leaders at midsize companies (500 to 3000 employees). The upside: 61% of those surveyed think they’ll be spending more on IT next year – is this bullish thinking about the economy or how much their own business (rev) will be growing?

Bill Snyder calls Dell “Bozo of the Month” for trying to trademark “cloud computing”. Yikes. Maybe not a “bozo” move but certainly inadvisable given how ubiquitous the term is. Here’s our take on it.

He’s a bit indignant that so much energy goes to sporting events like the Olympics rather than more important news that isn’t getting reported around the world.

This is in a year when tons of journalists are getting laid off.

This is in a year when there are tons of stories around the world that aren’t getting reported on.

Could we take half of those photographers and send them to Russia, for instance

Reminds me of a feeling I had back in college as an undergrad student studying social sciences and humanities, about the way my friends who were physicists interacted with the world. They were so awed by the stars, Mars, astrophysics, and it seemed to me interesting but altogether unimportant. They argued they may find something outside our planet that could help solve Earth-bound problems like disease, or find the origins of earth and humanity — but really they were doing it because they loved it. One of my friends had a good argument, though — there are enough people right now that we can specialize in what we care about, and there will still be others covering other topics. He could be a physicist and look into the universe’s origin, while I studied social interaction and writing, and our other friends looked into solving cancer or eradicating invasive plants in the native wetlands. We have to specialize, and there are enough of us to do it too.

I think it’s the same way in journalism — whether it’s sports, celebrity journalism, or coverage of politics and war, there are a lot of opportunities right now for journalists. Of course the business model is changing, and some old-schoolers won’t know how to roll with that, but generations change slowly; we’re learning.

Also, the Olympics is seen as more than a sporting event, it’s also a symbol of world competition and cooperation too — a way for countries to come together and share entertainment globally. I think that’s worth covering.

In the second post, Robert Scoble says there are plenty of great journalists but the public doesn’t care. In some ways I have to agree with that, but I don’t think it’s negative, necessarily. I had a conversation with someone the other day about world news reportage. He says, “I was just reading this story, but what does it matter to me if there’s a flood in some city in another country I’ll never visit and some farmer lost his sheep?” World news is only important when it’s relevant, so it’s no wonder that many people don’t care — if they don’t know much about the area, and it doesn’t affect them, they have no incentive to give it full attention. You can call that apathy, but I think it’s an important selectivity skill that humans have. We have to choose what to give priority to, so if nothing stands out as being particularly important, we just ignore it or gloss over it. Human nature…

Also I think the common person today just gets desensitized and doesn’t know where to turn their energy, when surrounded by so many crises. Either you focus on one specialty and do your best to work toward one cause in your life — and maybe that’s just in the course of your daily work — or you become a complete Attention-Deficit-Disorder case and bounce from one problem to the next, without knowing how to solve anything. That just causes a sense of bewilderment, despair, and either that bogs you down or eventually you get desensitized.

There’s a commenter on Scoble’s blog, Spencer, who talks about this generation’s apathy. There are so many people who want to blame today’s generation or the young generation for this “apathy” that they sense. But I see it as a survival mechanism that arises from the way information flows these days. We’re surrounded by crises, everyone wants us to know about them — the water shortage, global warming, death in Iraq, the national deficit. Okay, crisis, I get it. But no one gives a real clear idea on what any individual is really supposed to do to solve the problem. You can’t get involved with one global cause, without ignoring all the others, and if you do get involved it’s likely to become your life’s purpose. Most people are concerned with other things — their families, their work, personal development, their homes and futures, and really that’s enough to take up all their time.

I’m always amazed when I read about the early unionists. Emma Goldman for example, the activist who pushed for the 8-hr workday, and campaigned for free love in the early 1900s when women were still wearing corsets, used to work 16 hour factory days as a seamstress, then lead meetings late into the night. Today we lead cushy lives comparatively–8 hour days, plus commute and lunch, family time, dinner time, gym maybe, sleep… but it still doesn’t seem like we ever have enough energy and time.

What Emma had that most people today don’t, is a community living in the same conditions as herself, with clear goals about what they were campaigning for, and a cause that affected their own daily lives. Today, unionism and local activism is in much shorter supply, in part due to the many people who work fairly comfy desk jobs, and the problem that everyone has his own specialization, works in a cubicle, does his or her own thing. The problems we’re facing today in terms of global warming, global water shortage, aren’t the same kinds of problems that activists have fought for in the past, and there’s no clear road map for how to solve them. Our leaders sure aren’t leading the way.

What we do have, at least, is the Olympics, which is an age old symbol of international cooperation, play and competition…so, uh, go sports! As for full disclosure, I don’t actually have a TV and haven’t watched the Olympics in many years, but I do try taking short showers–does that help?

Q1: How do you handle variety of log sources? There are so many, almost beyond my capability.

A1: Sorry to ponder the meaning of "is" here, but what is meant by "handle"? It is really not that hard to collect logs from a large number of diverse sources (as long as the logs can be delivered via syslog or exist as files and can be collected). Now, there will certainly be challenges  when the volume of logs gets large, but if by "handle" you mean "collect + store", it is really not that hard, given the right tools. Now, if "handle" means "make sense of what all those logs are trying to tell you," it is a different story altogether.

Q2: You talked about the importance of logging; however for an intermediate or novice admin what are the starting steps .. what are the minimal logs they should start at once?

A2: Answered in "Log Management - Day 1" If you want a simple list of things to "enable today,"  I cannot really answer it since I know neither your needs, nor your environment. In other words, this is the "what is the meaning of life question?" :-)

Q3: What regulations, rules or guidance exist regarding sharing or visibility of logs to users?

A3: PCI DSS says in Requirement 10.5:  "Secure audit trails so they cannot be altered.
10.5.1 Limit viewing of audit trails to those with a job-related need
10.5.2 Protect audit trail files from unauthorized modifications
10.5.3 Promptly back-up audit trail files to a centralized log server or media that is difficult to
alter"

NIST guidance for FISMA also says something similar (for example, look in NIST 800-92 doc). Overall, log protection and security are mentioned in many other regulations as well.

Q4: Privileged groups membership monitoring in AD one of the most important from my point of view. However I did not find effective way to monitor/report on changes in those groups. Any recommendations?

A4: This is indeed a tricky one which might take more space to answer than I have here; it might also take you 'beyond logs.' One good source of information is Randy Smith's site and, specifically, his webinar on 'Active Directory "Logging Gap"' (here somewhere) - which covers how to audit things of that sort when then native logging is not sufficient.

Q5: How I can learn what exactly I need to log?

A5: OMG, this is a $1,000,000 question :-) Let me answer "how can I learn" part and not the "what exactly I need to log part,"  (also see discussion on "MUST-DO Logging for PCI?") as it is actually answerable. To learn what you need to log, first ask "Why?" (and then see this) - basically establish what you want to accomplish with logs, catalogue your systems, figure how to tweak the logging knobs - and then do it!

Q6: How granular should logging be? What is your recommendation for enterprise servers like domain servers and Windows servers?

A6: Again, too long to answer here in details (it will become a subject of a longer blog post later), but some pointers follow: here for Windows (MS site also have a few recommendations on audit policies)

Q7: What is "more control" and what is "less control" that you mention in the webcast? Can you give an example?

A7: OK, I did say that "sometimes when you implement more controls, you actually have less control." What do I mean? If you buy a firewall (a network security control) and then - over time, of course - configure it with 7800 rules (!) that are supposed to give you control over who can and cannot access your network, you will not gain control over your environment. You will actually be less in control of who is touching your network, compared to, say, having only 20 rules.

Q8: What about mandated NIST controls for government systems? Auditing is a specific control for Moderate and High risk systems. What list of events do you recommend for auditing?

A8: This is too long to answer here, but NIST 800-92 Guide is a really good source of such info ("Guide to Computer Security Log Management [PDF]") Also, see my presentation on NIST 800-92 Guide in the Real World.

Q9: The issue that many organizations get stuck on, is the monitoring process, and defining what exceptions to monitor for? Is there guidance / framework for this? How much of it is system specific and how much is applicable generally to all systems?

A9: I outlined some general ideas back in 2004 via this presentation (note to self - update that to be more 2008-relevant); it is mostly general, but also has pointers to specific system. Keep in mind that it is focused on security, not operational monitoring (which is often no less important - in fact, often MORE important)

Enjoy! Sorry for being brief with some of the answers - I am woefully late with this even as they are...

Other questions that I answered in the past:

• More Log Management Questions - Answered!
• Some Burning Logging Questions - Answered!

• Which problems will be solved and forgotten?
• Which ones will simply go away?
• Which ones will persist and in fact increase?
• Finally, which new ones might emerge?

First, let me bet my ass that "Not knowing what to log" problem will be licked in 18-24 months; at least as far as major regulations go, people will have a pretty good idea a) what  the auditors want them to log (and review!) b) what they need to log for solving their problems. Now, for esoteric log sources (and custom applications) might still present a challenge from that point of view, but for basic "staples" (firewall, network gear, major OS) the mystery will be over (again, see "Tell me EXACTLY what to log for PCI?"  for reference)

Next, the problem of "Log volume" will  definitely get worse, much worse.  One might think that 100,000 each second is a lot of log - but there WILL BE more at many companies! Big application log explosion is coming, fueled by the need to address logging in areas where such motivation was lacking before (basically, custom and vertical applications) as well as harness the power of "uncommon" logs for such tasks as fraud analysis or SOA monitoring. Keep in mind that even though in some areas logging is NOT a preferred way of monitoring and auditing activities (see this discussion on database logs here), application logging will still explode on us...

The problem of "Log diversity" (the fact that most logs all look different in format and meaning) will get worse before it will get better - and better it WILL (!!!) get since standards are being developed. We will see people struggling with all sorts bizarro log data in the coming years. Virtualization, web services and SOA, various ERP applications and even cloud services will increase the diversity of logging in the coming years.

Similar to the above, a problem of "Bad logs" (ones that are subjective, miss key information, require groping for a crystal ball to understand, turn log analysis into dark voodooistic experience or are useless in some other way) will also follow the pattern of the above log diversity problems - it will get worse before it gets better (via the CEE standard effort that now covers the OpenXDAS effort as well!) I noticed that people started asked me questions about "how to do application logging right?" and "what to tell application developers about logging?" which almost never happened in the past. BTW, watch my blog for some uber-fun info on that!

"Getting the logs"  has gotten much easier in recent years; agentless collectors like Project Lasso (which, BTW, just got updated) and grabbing  files remotely via secure protocols made application log collection easier (syslog-NG with TCP transfer and buffering also helped). Next, Windows 2008 will make it MUCH easier for the whole Windows kingdom due to their use of web services (thanks Eric!). However, in the future it might resurface as we try to collect logs from "weird" places, again, clouds come to mind as well as virtual environments (e.g. how do you get logs off a dormant VM?). What's the next frontier in this area? Log discovery - automatic finding and identifying log files on systems in order to analyze and retain them (Yo, my t-shirt-making colleagues... :-))

All this, however, pales in comparison with my favorite "uber-challenge", "Making sense of logs in  an automated fashion" - this baby is definitely not going away in 2-3 years. Much more research is needed to make that "log->conclusion" jump automatically without head-scratching, invoking ancient deities and cursing under ones's breath. Only then we can attempt to reliable handle "proactive logging" (i.e. analyzing various failure or compromise precursors in logs and then predicting the future based on them), another Holy Grail of logging domain.

Anything new will emerge? Yes, I think awareness of the "Logging Gap" problem will grow. "Logging gap" happens when you combine "a need to log" with utter "inability to do so."  For example, this will happen when people will know that they HAVE TO log, say, for compliance, but will have no way of doing it due to application or platform limitations. This will become one of the challenges and special "logging add-ons" will appear to close the logging gap and create additional logs where activity audit is desperately needed, but native logging is not helping to achieve it.

Also, I think people will finally wake up to "Log security" challenges - i.e. producing for use as evidence, compliance attestations, etc. Log security is not getting the attention it deserves, but I think this challenge will finally emerge in full force in the next 2-3 years. My next poll will address that :-)

Anything else I missed? Share away!

Related posts:

• Ideal Tool to Solve Real Problems ... of the Near Future?

• Ideal Log Management Tool?