for final project you will develop an automated web crawler to collect
For final project, you will develop an automated web crawler to collect data from a website of your
choice, store them in a database, and perform data analytics using Python.
1) First, identify a website as your data source, then identify target data fields your team plans to
collect. You should aim to collect as much data as possible, even if you do not initially expect to use
some data fields for analysis. This is because retroactive collection could be time-consuming if you
find that you are missing some needed data later on.
2) Set up a database to store your data.
You can use any database, including sqlite3, MS SQL Server, MySQL, MongoDB, etc. But note
that Excel or a csv file is not a database.
Based on the data fields you identified from the website, design and create one or more
tables that host your dataset to be collected.
c) After finalizing your database tables, develop the web crawler so that it directly inserts data
into your database (instead of, for example, downloading files as a csv file first, then
importing the csv file into the database).
3) Based on collected data in the database, perform some analyses to obtain insights. The types of
analyses can include at least two of the following (but not limited to):
a) Descriptive analysis
d) Sentiment analysis
e) Other text mining analysis
4) Present your work in a video presentation.
*The amount will be in form of wallet points that you can redeem to pay upto 10% of the price for any assignment. **Use of solution provided by us for unfair practice like cheating will result in action from our end which may include permanent termination of the defaulter’s account.Disclaimer:The website contains certain images which are not owned by the company/ website. Such images are used for indicative purposes only and is a third-party content. All credits go to its rightful owner including its copyright owner. It is also clarified that the use of any photograph on the website including the use of any photograph of any educational institute/ university is not intended to suggest any association, relationship, or sponsorship whatsoever between the company and the said educational institute/ university. Any such use is for representative purposes only and all intellectual property rights belong to the respective owners.