Mini Project Report

On

“Analyzing Web log data using Hadoop”

Submitted in partial fulfillment of the

Requirements for the award of the degree of

 

Bachelor of Technology

In

Computer Science & Engineering

By

G.VEERA REDDY (14R21A05J9)

 

Under the guidance of

Mr. K. SAI PRASAD,

Assistant Professor, Department of CSE.

 

 

 

Department of Computer Science & Engineering

 

 

 

2017

 

 

 

Department of Computer Science & Engineering

 

 

 

                              CERTIFICATE

This is to certify that the project entitled “Analyzing web log data using Hadoop” has been submitted by G Veera Reddy (14R21A05J9) in the partial fulfilment of the requirements for the award of degree of Bachelor of Technology in Computer Science and Engineering from Jawaharlal Nehru Technological University, Hyderabad. The results embodied in this project have not been submitted to any other University or Institution for the award of any degree.

 

 

 

 

 

 

 

 

Internal Guide                                                                               Head of the Department

 

 

 

 

 

 

                                                 External Examiner

 

DECLARATION

 

I hereby declare that the project entitled “Analyzing web log data using hadoop” is the work done during the period from June 2017 to December 2017 and is submitted in the partial fulfillment of the requirements for the award of degree of Bachelor of technology in Computer Science and Engineering from Jawaharlal Nehru Technology University, Hyderabad. The results embodied in this project have not been submitted to any other University or Institution for the award of any degree or diploma.

 

 

 

 

 

                           

                                                                                                                                 

 

 

 

   G Veera Reddy

                                                                                                                            (14R21A05J9)

 

 

 

 

 

 

 

 

 

 

        

 

 

 

 

ACKNOWLEDGEMENT

 

 

There are many people who helped me directly and indirectly to complete my project successfully. I would like to take this opportunity to thank one and all.

 

First of all I would like to express my deep gratitude towards my internal guide K SAI PRASAD – Assistant Professor, Department of CSE for his/her support in the completion of my dissertation. I wish to express my sincere thanks to N. CHANDRA SEKHAR REDDY, HOD, Dept. of CSE and also to principal Dr. P BHASKARA REDDY for providing the facilities to complete the dissertation.

 

I would like to thank all our faculty, coordinators and friends for their help and constructive criticism during the project period. Finally, I am very much indebted to our parents for their moral support and encouragement to achieve goals.

 

 

 

 

 

 

                                                                                                                               G Veera Reddy

                                                                                                                                (14R21A05J9)

Abstract:

In the contemporary world, data analysis is a challenge in the era of varied inter-disciplines, though there is a specialization in the respective disciplines. In other words, effective data analytics helps in analyzing the data of any business system. But it is the big data which helps and accelerates the process of analysis of data paving way for a success of any business intelligence system. With the expansion of the industry, the data of the industry also expands. Then, it is increasingly difficult to handle huge amount of data that gets generated no matter what’s the business is like, range of fields from social media to finance, flight data, environment and health.

 Big Data can be used to assess risk in the insurance industry and to track reactions to products in real time. Big Data is also used to monitor things as diverse as wave movements, flight data, traffic data, financial transactions, health and crime. The challenge of Big Data is how to use it to create something that is value to the user. How can it be gathered, stored, processed and analyzed it to turn the raw data information to support decision making. In this project Big Data is depicted in a form of case study for analyzing web log data using hadoop.

Log files are often very large and can have complex structure. Although the process of generating log files is quite simple and straightforward. This often leads to a common situation, when log files are continuously generated and occupy valuable space on storage devices, but nobody uses them and utilizes enclosed information. This can analyze different kinds of log files such as- Email logs, Web logs, Firewall logs Server logs, Call data logs.

 

 

 

 

 

 

 

CONTENTS

 

Certification                                                                                                                           i

Declaration                                                                                                                            ii

Acknowledgement                                                                                                                 iii

Abstract                                                                                                                                 iv

1. Introduction

1.1 Bigdata-3V’s                                                                                                        8

1.2 Ecosystem                                                                                                             9

1.3 Applications of bigdata                                                                                        10

1.4 Hue                                                                                                                       11

2. Literature Survey                                                                                                                                     2.1 Existing System                                                                                                          12

2.2 Proposed System                                                                                                  13

3. Requirement Analysis                                                                                                      14

3.1 Hardware Requirements

3.2 Software Requirements

4. Implementation                                                                                                                 15

4.1 Problem Definition

4.2 System Architecture

5.Methodology

5.1 HDFS                                                                                                                    16       

5.2 Map Reduce                                                                                                          19

5.3 Hue                                                                                                                       21

6. Screen Shots                                                                                                                      22

7.Limitations and Future Enhancement                                                                             26

8.Conclusion                                                                                                                          28

Bibliography