Hello guys, today in this tutorial I will introduce to a web scraping library called
as Puppeteer and I will automate the process of scraping the information from github.com
such as the username of the person, biography, number of public repositories which are available
all this information we will be scraping from github and without any API
So we will be using Puppeteer which is a scraping library or you may say it is an automation
tool for nodejs and javascript developers. It's a famous library, we will be using it and we will be saving all this information
in a comma separated value file which is a csv file. Let me show you the demo here
So I have written this script right here and now if I execute this script it will go to
my profile nodeindex.js, I am passing my username as an argument and it will now open the browser
automatically, it will go to my profile and now in the console you will basically see
the profile information scraped. This is a username gauthi123456, this is a biography which is scraped from my github
account, public account and the number of repositories 72 and data saved to githubprofiles.csv
Scraping complete and if I see in the left hand side a file is created github underscore
profiles.csv and now if I try to open this file you will basically see these are username
biography repositories. So all this information is scraped right here guys, you will see 72, this is my biography
you will see, you can cross check as well by going to github.com slash this username
You can input any username here, basically this is scraping this information, we are
scraping this username, this biography and then we are scraping the number of public repositories
A total of 154 is there but the public repositories that I shared is 72 and apart from that we
have private as well but this information we have scraped and you can pass any username
guys, any username, let me pass brad privacy, this is, let me pass it here
So you will see now number of repositories 273, full stack web developer and online instructor
and if I check, you will see it will get added in the CSV file, the second one, entry brad
privacy, full stack web developer, instructor, specializing and mostly js. So it's an automation script guys, you can make this and scrape information from github automatically
Let me cross check, so this is basically the thing guys, you will see that
So let me now show you guys how I build this, so for this we are using Puppeteer which is
a nodejs library, the command is simple, npm i puppeteer, just execute this and weekly
downloads, 3 million weekly downloads are there. So once you install this library, it's very simple, npm i puppeteer, just inside this
we need to require this package at the top and then guys we will basically make a function
async function which will say scrape github profile, scrape github user profile and here
you will pass only the username, so you need to pass the username and this username you
can even take dynamically using the browser inside express js, react, angular, you can
make a frontend and consume the backend API and then you will open this, automatically
this puppeteer library, you will launch this and here we need to pass the headless option
if you want to see the browser opening you will pass here false, if you want to do this
in background then it will do true, let me do it in the background, I don't need to see
the browser opening, so then we will open the new page using this method and then we
will do all our things in the try-catch block, so if any sort of error take place, we will
simply console.log error and now we just need to navigate to that user profile page, so
we will go to that page by using page.goto, goto is a method where you will pass the URL
so in order to go to a specific URL or github user profile, github.com and followed by the
username that is passed, so we are using template string right here and this will go to that
page and then we need to wait for all the selectors to be loaded, so wait for selector
and then we need to wait for this selector which is having this class vcard-username
so after this selector is loaded we will now be extracting the user information, so first
of all we will extract the user profile, so we will say page.evaluate and here we will
actually pass a function arrow function which will extract the username of the person by
using very simple query selector which is a dom method which you use in vanilla javascript
we will basically target a element which is having this class vcard-username and we will
actually extract the content by using text content property and then we will trim the
content, we will trim the white space which is there if any and then we will actually scrape
the biography of the person by again using query selector and this time we will select this class
having element user profile bio, so this is accessible in the browser guys, you can cross
check all these elements have a particular class attached to it, so in this way you can
scrape anything from internet, these are the fundamentals of web scraping and after scraping
biography we will scrape the number of public repositories, repo count and query selector and
counter, this is a class which is having this value and again we will trim the value like this
so here user profile bio, so here we will simply check guys basically we will extract the biography
bio element, if the biography is there in that case you will say bio element dot text content
dot trim, if the biography is not available in that case you will use this ternary operator like
this colon we will say no biography available, if the person doesn't contain a biography then we
will simply say this text no biography available like this and then we will return these three
things, we will in an object here we will return an object from this function, first is the username
of the person then we will return the biography and then we lastly we will return the repo count
these three variables we are returning from this function that's all and after that we
will simply console log user profile info information and here we can even add a finally clause as well, finally clause as well we can add and here
we can basically close the browser after we do all the stuff browser dot close, we will close
the browser and now we can simply call this function and where we can pass the username
so let's suppose I pass my own username kothi123456 and then it returns a promise guys because here
we are returning a object from this asynchronous function, asynchronous function returns a promise
guys so we can handle this using dot then and here we can simply say scraping complete
so if you do this we can console log this information if you see if you try to console
log this information in the right here if I try to console log this information right here so
user profile information username biography number of repository so if I try to run this so now you
what you will see it will run it in the background so node index.js it will run it in the background
the browser will not show and because we are passing this headless option to true so it will
work in in the background and then it will show this information in the console the user profile
information username biography number of repositories which is 72 so now we need to basically store this
information in the CSV file so how we can do that it's pretty easy guys how we will do this we will
actually create a this line of code we need to simply write let me just use chart gpt for that
chart gpt is a very handy tool guys where you can simply get some nice little code so here we will
declare a CSV data variable and we will say here user profile dot username and here we are
defining basically the headers so what headers should be there so they should be separated by
comma so user profile dot biography comma and then lastly user profile dot repo count so we have
defined the CSV data we just need to put a newline character because it should be the data should
come on the newline character so these are defined the rows the header rows which is username
biography and repo count after that we just need to append by using the file system module which is
available for us as a built-in module so require the file system module of nodejs and it contains
a method called as append file and append file we will basically give it a name which I will
give it as user profile dot username dot CSV and this is the name of the file and we will append
this data CSV data and UTF-8 and that's all this is all that you need to do guys
if I try to run this application node index.js so it will take some time and after that it will
create a file called as once it's completed you will see basically the CB argument must be of
types function receive type of UTF-8 let me see guys what is the problem let me check here okay
okay sorry okay this needs to be a wait I forgot to
okay I forgot to do this we do need to write this code guys just right before right here after we do this just write
this code here we are simply comparing that if this file already exists in that case let me copy
this code first of all paste it here and then explain to you what is happening so we are
basically scraping for this username we are passing it as an argument and right inside here we are
creating a new CSV file by the name of github profiles.csv you can change this name accordingly
let's suppose I need to give it a dynamic name I can give it as user profile dot username dot CSV
so whatever is the name of the user will be saved as a name of the file so here we are three rows
three header rows which is the username biography and retro count this information we are storing
in this file and here we are simply checking that if the CSV file already exists then we need to
simply add the headers so if the file doesn't exist in that case we just need to add these
headers which is these headers we are adding username biography repositories so if this is
the first record then we are basically adding the headers for the very first time that's all
and now if I try to run this application guys so what it will say on the left hand side it will
create gothi123456 whatever is the username and you will basically see this is my thing here
gothi123456 and the header rows are not added because the name of the file was different so
let me here also make the dynamic or let me say data.csv in both the places data.csv
and close all this so now run this data.csv oh sorry I think I made a mistake here
this needs to be data.csv not github.profiles so delete this so once again run this you will
now see this data.csv will created and these are your header rows username biography repositories
so we have this information added you will see let me take another example let me change this
username to brad traversy so this will append information in this file a new entry will be
added you will see that the second entry is added a new file is not created it will append right
here so in this way you can automate this process guys you can scrape as many public profiles on
github using the script of Puppeteer in nodejs so please hit that like button subscribe the
channel and I will be seeing you in the next video