HYBRID Could Log ANALYSIS-LOG_DND(SP) – DEVELOPPARADISE
17/07/2018

HYBRID Could Log ANALYSIS-LOG_DND(SP)

Introduction

As a one-stop service for log data, Log Service (Log for short) experiences massive big data scenarios of several hybrid cloud services provider. Log Service allows you to quickly complete the collection, consumption, shipping, query, and analysis of log data without the need for development, which improves the Operation & Maintenance (O&M) efficiency and the operational efficiency and builds the processing capabilities to handle massive logs in the DT (data technology) era.

HYBRID Could Log ANALYSIS-LOG_DND(SP)

What is Hybrid Cloud?

A combination of one or more public and private cloud environments is known as hybrid cloud. It’s allows users to access the resources on-demand through self-service portals supported by automatic scaling and dynamic resource allocation. Hybrid cloud is a pool of virtual resources—developed partially from hardware owned and managed by a third-party company as well as hardware owned by the enterprise using the cloud—orchestrated by management and automation software for the user’s access.

An encrypted application programming interfaces (APIs) that help transmit resources and workloads, while the public and private cloud environments that make up a hybrid cloud remain unique and separate entities, migrating between them is facilitated by . This separate—yet connected—architecture is what allows enterprises to run critical workloads in the private cloud, less sensitive workloads in the public cloud, and pull resources from either environment as desired. It’s an arrangement that minimizes data exposure and allows enterprises to customize a scalable, flexible, and secure portfolio of IT resources and services.

Now a day’s hybrid cloud environments can include on-premise infrastructure, traditional virtualization, bare-metal servers, and containers. Furthermore, the hybrid cloud may be made up of multiple public clouds hosted by more than one provider.

Hybrid Cloud Architecture

Hybrid cloud architecture is the integration of on-premises resources with cloud resources. For most organizations with on-premises technology investments, operating in a hybrid architecture is a necessary part of cloud adoption. Migrating legacy IT systems takes time. Therefore, selecting a cloud provider who can help you implement a thoughtful hybrid strategy, without requiring costly new investments in on-premises hardware and software, is important to simplify operations and more easily achieve your business goals.

There are many strategic partnerships available with long time leaders in on-premises platform providers such as VMware, Intel, Microsoft, SAP, and others to allow you to run your existing enterprise applications with full support and high performance.

An Architectural Diagram of Hybrid Cloud

A simple network diagram for a hybrid cloud environment and maps some of the key areas that should be addressed by leveraging a defence in depth approach to enhance the overall security of this solution.

Diagram – 1

HYBRID Could Log ANALYSIS-LOG_DND(SP)

Source: blogs.technet.microsoft.com

The areas mapped on this diagram reflect the core components in a hybrid cloud scenario. Each core area must be expanded in order to identify the threats and vulnerabilities that are applicable to it. The numbers on this diagram do not reflect any priority order; it only organizes the six major components.

Log Operations in Hybrid Cloud

The best way to monitor the health of your IT system is perhaps to dive deeper into the system logs. Organizations generate tons of data everyday just in the form of logs, carrying information on alerts and notifications generated at different points of time. In fact, a large deployment of mail server, such as Microsoft Exchange generates 1 GB worth of logs every day! It is imperative that organizations store and analyze log data to gain business insights, along with adherence to local laws and regulations. A simple diagram of a hybrid cloud log life cycle is given below:

Diagram – 2

HYBRID Could Log ANALYSIS-LOG_DND(SP)

Source: corestack.io

Mostly, functions are small and need to execute in less amount of time in a stateless environment generally in serverless applications. Logging then becomes a challenge for DevOps to do Root Cause Analysis (RCA) or get insights about the behavior of their applications.

One logging approach is to add code to your functions to send logs to an external API, which can be a painful process for you and the performance of your applications. You don’t want to add complexity to your functions in terms of security, extra endpoints and protocols, as opposed to just concentrating on your business logic scoped to the single function. Because every millisecond counts when using serverless, you don’t want end users waiting while you send logs during action invocation.

Implement Log Management

IT teams can more effectively detect and alert on any unauthorized activity or security threats to their network by automating the collection, storage and back-up of logs. Some examples of log collection include:

  • Server and application logs
  • Security logs from firewalls
  • DDoS and intrusion prevention systems
  • Syslogs from any source
  • IIS web server logs
  • Secure file server logs

Be sure to track, alert and report on events like access and permission changes to Files, Folders and Objects, and always collect the most common log types such as Syslog, Microsoft event or W3C/IIS to help identify potential threats to your network.

With automated log collection and analysis, IT teams can proactively detect any unusual activity on their network and immediately rectify the situation. Additionally, they can leverage log management data to produce monthly reports to provide evidence for audit and compliance purposes at the corporate or executive level.

Real-time log collection and consumption (LogHub)

Functions:

  • Use Elastic Compute Service (ECS), containers, mobile terminals, open-source software’s, and JS to access real-time log data (such as Metric, Event, BinLog, TextLog, and Click data).
  • A real-time consumption interface is provided to interconnect with real-time computing and service.

Purposes: ETL, Stream Compute, monitoring and alarm, machine learning, and iterative computing.

LogShipper

Stable and reliable log shipping ships LogHub data to storage services for storage and big data analysis. Supports various storage methods such as compression, user-defined partitions, row storage, and column storage.

Purposes: Data warehouse + data analysis, audit, recommendation system, and user profiling.

  • Query and real-time analysis (Search/Analytics)
  • Index, query, and analyze data in real time
  • Query: Keyword, fuzzy match, context, and range
  • Statistics: Rich query methods such as SQL aggregation
  • Visualization: Dashboard and report functions
  • Interconnection: Grafana and JDBC/SQL92

Purposes: DevOps/online O&M, real-time log data analysis, security diagnosis and analysis, and operation and customer service systems. A simple diagram of real time hybrid cloud data log analysis is given below:

Diagram – 3

HYBRID Could Log ANALYSIS-LOG_DND(SP)

Source: static-aliyun-doc.oss-cn-hangzhou.aliyuncs.com

Security Risks

The benefits of hybrid cloud infrastructures are undeniable. Such environments can provide companies with impressive scalability, availability and efficiency, while also offering significant cost savings.

However, the security risks hybrid environments can introduce need to be taken seriously, as even the smallest network vulnerability can lead to dire consequences.

Conclusion

I tried to cover the hybrid cloud infrastructures, log analysis, etc., and I hope you guys enjoy this article.

History

  • 16th July, 2018: Initial post