Welcome to the 17th issue of our monthly Newsletter “Guardian – News & Views,” an endeavour to spread awareness among industry leaders on best practices and emerging trends in the information security space through discussions.
The critical role that log analysis plays in operational IT security is often overlooked. Log analysis is essentially the monitoring and analysis of data (log files) generated by network devices, operating systems, applications, etc. in order to extract information about their functioning and integrity.
While log analysis serves a number of purposes – performance monitoring, evaluating system changes, analysing the problems that are logged into the system – of greater importance is pure security log analysis which handles specific security incidents and events recorded in the log files.
Security log analysis helps you protect and secure your network by compiling log data from various operating systems, applications and devices. This data can then be used to proactively detect and prevent disaster, as well as identify the cause of any security incident that does occur.
The need for security log analysis is driven by a variety of factors:
Compliance with security policies:
Different regulatory guidelines (RBI Guidelines, ISO 20001, IT Amendment Act 2009) clearly state that organizations need to enable log analysis that performs to a certain requisite standard.
Forensic requirements:
In cases of organizational espionage, organizations look to build a case by identifying the attacker and compiling evidence that is admissible in a court of law. The exhaustive evidence needed for a security incident can only be gleaned from log analysis. Forensic analysis to aid in investigations requires the system to log different kinds of activities and generate meaningful information. An expert with the requisite domain knowledge then needs to analyze those logs accurately and report back to the concerned authorities.
Threat detection and security incident response:
Since logs provide information on what is happening in the system at any given time, unexpected discrepancies can help in detecting threats. Logs also provide data on the nature of attack and the extent to which the system was compromised; this assists in responding appropriately to a security incident and in mitigating the problem.
Correlating data logs from different sources can lead to an integrated view of security threats and their resolution
An emerging trend in this domain is the attempt to usefully correlate and process the contents of multiple logs. The need of the hour is for correlated context-based log analysis. The diverse nature of IT infrastructure today – a variety of applications, operating systems, etc. – demands a consolidated view of data. In traditional silo-based log analysis, data is available on the specific device being targeted, but this is of no real relevance given that an attacker targets a system by bypassing multiple gateways. Hence the need for logs that generate records on different gateways and the path that the attacker takes to compromise the end system. This will lead to a complete understanding of the security threat and help in resolving it more effectively.
To strengthen this process, organizations are also using application and content-aware log analysis which represents a new generation of log analysis capabilities that extend the value and benefits of log analysis by providing visibility into the contents of applications, documents and protocols. This in-depth data provides sufficient context and content to make informed decisions.
Data logging is only as relevant as its interpretation
As more and more firewalls, anti-virus solutions, intrusion detection systems and security devices generate copious amounts of data, the effort is on to compile and categorise all this data in a manner appropriate to effective analysis. It is not enough to simply have the requisite logging infrastructure in place – what is more critical is the interpretation of those logs. This means that even before you set up your logging infrastructure you need to have a clear plan as to the purpose of the logs, what you want to log and how you intend to compile log data, monitor and review the logs, and then translate the data into actionable results. Organizations need to additionally invest in resources to monitor and analyze generated logs. This includes identifying domain experts with the technical ability to review the logs and actually understand what they indicate.
Most log analysis is retrospective. While this approach works well for certain activities such as identifying the bottlenecks in the efficient performance of a system, tracking traffic patterns, etc., it is not ideal from a security perspective. It is not sufficient to only review logs after a major incident; you need to constantly monitor logs in order to anticipate a threat. Monitoring the system for failures and identifying security breaches calls for real-time analysis. While practically speaking, there is no technology available today that can actually perform real-time analysis of logs, constant monitoring of logs will allow for a near real-time approach where you can be alerted as an attack is happening and take corrective measures to prevent the system from being compromised.
Best practices to make security log analysis more effective
Data logging should be done in an intelligent manner so as to generate data that is both comprehensive and relevant. Too much of data on every operational aspect is akin to having no data at all. In essence, log files should record the moments when critical information assets are accessed (by whom, when, etc). Key business processes such as data backups, system configurations, database transactions, etc. should also be recorded. Finally, given the range of access that is available to administrators, logs should record when administrator accounts access critical information. In terms of managing log files, some best practices that ensure that the effective functioning of security infrastructure include storing log files on dedicated log management systems.
Security log analysis can generate a wealth of data critical to securing the information and assets of an organization. But if not reviewed on a continuous basis and interpreted expertly, the data becomes merely copious amounts of irrelevant records that serve no purpose. In this context, the call is for a correlated interpretation of data logs from various sources to build a holistic and complete knowledge base of the security infrastructure.
Economics of security
For some time now, we have been talking about the need for a proactive approach to security, and its effectiveness and benefits in managing risks; for instance, it is always more effective and economical to build secure software rather than testing and fixing it after development or in production. In fact we even learned this basic fact in school with lessons which said “Prevention is better than cure”.
In risk management we come across controls which are:
Preventive: Controls which ensure that exposure do not or cannot occur
Detective: Controls which help us capture exposures if they happen or are happening
Corrective: Controls which enable us to correct exposures
Nowadays there is lot of focus on Detective controls which includes the deployment of technology solutions which detect and capture unwanted network activity, access attempts, patterns etc. Needless to say these investments and focus are needed, but we need to move our risk management posture more towards the preventive side. We must do more to ensure we do not have weak areas which can be exploited.
I recently read the book “Superfreakonomics” by Steven Levitt and Stephen Dubner. I am usually apprehensive of sequels as they never match up to the original, but luckily this book was a good read. I came across two examples in the book which illustrate the point about preventive controls.
After the 7/7 terrorist attacks in London a team was formed to use statistical information to identify terrorists. Data points used to identify suspects were banking usage patterns such as:
They make large deposits in cash and withdraw small amounts
PO boxes are used as addresses and they often change these
There are regular wire transfers to other countries but always below the threshold for bank triggering requirements
They never use savings accounts or fixed deposits even though the account has idle money
Transactions do not show normal living expenses and regular out flows such as insurance payments etc.
As one can imagine it would be difficult to come up with an algorithm to make the system accurate. Let us say that a system is developed with 99% accuracy and that there are 500 terrorists in the UK; 495 of them would be identified which would be great. The problem is that with 50 million adults living in the UK the system would also wrongly identify 1% of them which is 500,000 people. This would be huge problem to manage and is similar to the “False positive” issue in the information risk management world. Hence the best detective control system or technology would always have a false positive issue which would significantly reduce the benefits from the system.
Another example is the detective control deployed at airports which require us to remove our shoes at the security check / scan. This started after a Richard Reid tried to ignite a shoe bomb; fortunately he failed, but statistically he succeeds in killing the equivalent of 14 lives a year in the US! How?
Let us say it takes, on an average, one minute to remove and replace the shoes in the airport security line. In the US, this happens about 560 million times a year, which is equal to 1,065 years. Average US live expectancy is 77.8 years, which yields a total of 14 person-lives a year.
The above examples may sound dramatic (statistics and economics can be used to communicate any message depending on what you want to say!). However the underlying theme makes sense which is that we have to focus on a proactive approach to security in order to be more effective and economical.
