{"id":9175,"date":"2020-03-18T15:52:20","date_gmt":"2020-03-18T10:22:20","guid":{"rendered":"https:\/\/www.botreetechnologies.com\/blog\/?p=9175"},"modified":"2024-01-18T18:33:35","modified_gmt":"2024-01-18T13:03:35","slug":"how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails","status":"publish","type":"post","link":"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/","title":{"rendered":"Steps for web scraping with Ruby on Rails"},"content":{"rendered":"<h2>What is Web Scraping?<\/h2>\n<p><a href=\"https:\/\/www.botreetechnologies.com\/blog\/web-scrapping-using-python\/\" target=\"_blank\" rel=\"noopener noreferrer\">Web scraping<\/a> is a technique to extract a large amount of data from a website and display it or store it in a file for further use.<\/p>\n<p><span style=\"font-weight: 400;\">Often, companies need to extract volumes of data from a particular site to process and analyze. Web scraping with Ruby on Rails is an easy way to do that.<\/span><\/p>\n<p>It is used to crawl and extract the required data from a static website or a JS rendered website.<\/p>\n<p>You can also use the <a href=\"https:\/\/www.zenrows.com\/\" target=\"_blank\" rel=\"noopener\">best web scraping API tool<\/a> to integrate data harvesting techniques directly with your Ruby on Rails project. The choice you make here all comes down to the type of scraping you want to carry out, the extent of your coding skills, and the degree to which you are hoping to automate the processes that are involved.<\/p>\n<p><b>When talking about web scraping using Ruby, here are a few terms to get familiar<\/b><b> with:<\/b><\/p>\n<h3>Few terms to get familiar with:<\/h3>\n<ul>\n<li><a href=\"https:\/\/github.com\/sparklemotion\/nokogiri\" target=\"_blank\" rel=\"noopener noreferrer\">Nokogiri<\/a>:\n<ul>\n<li><span style=\"font-weight: 400;\">A gem for web scraping Ruby, Nokogiri, <\/span><span style=\"font-weight: 400;\">uses CSS selectors or XPath for web scraping.<\/span><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/github.com\/teamcapybara\/capybara\" target=\"_blank\" rel=\"noopener noreferrer\">Capybara<\/a>:\n<ul>\n<li>Allows JS-based interaction with the websites.<\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/github.com\/vifreefly\/kimuraframework#pipelines-send_item-method\" target=\"_blank\" rel=\"noopener noreferrer\">Kimurai<\/a>:\n<ul>\n<li><span style=\"font-weight: 400;\">It is a framework for web scraping with Ruby.<\/span><\/li>\n<li>Combination of Nokogiri + Capybara.<\/li>\n<li>Allows scraping data for JS rendered websites and even static HTTP requests.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>There are few tools available for web scrapings such as Nokogiri, Capybara and Kimurai. But, <strong>Kimurai is the most powerful framework to scrape data.<\/strong><\/p>\n<h3>Kimurai<\/h3>\n<p>A web scraping framework in ruby works out of the box with Headless Chromium\/Firefox, PhantomJS, or simple HTTP requests and allows us to scrape and interact with JavaScript rendered websites. <span style=\"font-weight: 400;\">t also aligns with any Ruby web scraping library for better functionalities.<\/span><\/p>\n<p><strong>Features :<\/strong><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\">Ruby web scraping for Javascript websites.<\/span><\/li>\n<li>Supports Headless Chrome, Headless Firefox, PhantomJS or Simple HTTP requests(mechanize) engines.<\/li>\n<li>Capybara methods used to fetch data.<\/li>\n<li>Rich library for built-in helpers to make scraping easy.<\/li>\n<li>Parallel Scrapping &#8211; Process web pages concurrently.<\/li>\n<li>Pipelines: To organize and store data in one place for processing all spiders.<\/li>\n<\/ul>\n<p><strong>You can also scrape data from JS rendered websites, i.e. infinite scrollable websites and even static websites. Amazing right\u00a0 !!!\u00a0<\/strong><\/p>\n<blockquote><p><strong>Read Also: <a href=\"https:\/\/www.botreetechnologies.com\/blog\/web-scraping-using-mechanize-in-ruby-on-rails\/\" target=\"_blank\" rel=\"noopener noreferrer\">Web scraping using Mechanize in Ruby on Rails<\/a><\/strong><\/p><\/blockquote>\n<h3>Static Websites:<\/h3>\n<p>You can use this framework in 2 ways:<\/p>\n<ol>\n<li>Making a rails app and extract information with the help of models and controllers.\n<ul>\n<li>Create a new rails app.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<p style=\"padding-left: 80px;\"><code>rails _5.2.3_ new web_scrapping_demo --database=postgresql<\/code><\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>Change the database configurations in app\/config\/database.yml as per the requirement to run in the development environment.<\/li>\n<li>Open rails console and create a database for the web app:<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p style=\"padding-left: 80px;\"><code>rails db:create<\/code><\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>Add gem &#8216;kimurai&#8217; to Gemfile.<\/li>\n<li>Install the dependencies using:<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p style=\"padding-left: 80px;\"><code>bundle install<\/code><\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>Generate a model using the below command with the parent as <strong>Kimurai::Base<\/strong> instead of <strong>ApplicationRecord:<\/strong><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p style=\"padding-left: 80px;\"><code>rails g model Web Scrapper --parent Kimurai::Base<\/code><\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>Perform database migrations for this generated model.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p style=\"padding-left: 80px;\"><code>rails db:migrate<\/code><\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>Generate a controller using:<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p style=\"padding-left: 80px;\"><code>rails g controller WebScrappersController index<\/code><\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>Make a root path for the index action:<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p style=\"padding-left: 80px;\"><code>root 'web_scrappers#new'<\/code><\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>Add routes for WebScrapper model:<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p style=\"padding-left: 80px;\"><code>resources: web_scrapper<\/code><\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>Add a link to the index.html.erb file as shown below:<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p style=\"padding-left: 80px;\"><code>&lt;%= link_to 'Start Scrap', new_web_scrapper_path %&gt;<\/code><\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>Now add an action in the WebScrappersController to perform scraping:<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p style=\"padding-left: 80px;\"><code>def new<\/code><br \/>\n<code>Web Scrapper.crawl!<\/code><br \/>\n<code>end<\/code><\/p>\n<p><strong>Note:\u00a0<\/strong> Here, <strong>crawl!<\/strong>\u00a0 Performs the full run of the spider. parse method is very important and should be present in every spider. <strong>The entry point of any spider is parse.<\/strong><\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>Now add some website configurations in the model for which you need to perform scrapping.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><script src=\"https:\/\/gist.github.com\/HunnyJummani\/458bfbc1968dd61161e85494211ddaa4.js\"><\/script><\/p>\n<p>Here,<\/p>\n<p style=\"padding-left: 80px;\"><code>@name<\/code> = name of the spider\/web scraper<\/p>\n<p style=\"padding-left: 80px;\"><code>@engine<\/code> = specifies the supported engine<\/p>\n<p style=\"padding-left: 80px;\"><code>@start_url<\/code>s = array of start URLs to process one by one inside parse method.<\/p>\n<p style=\"padding-left: 80px;\"><code>@config<\/code> = optional, can provide various custom configurations such as user_agent, delay, etc\u2026<\/p>\n<blockquote><p><strong>Read the Case Study about &#8211; <a href=\"https:\/\/www.botreetechnologies.com\/case-studies\/rpa-web-scraping\" target=\"_blank\" rel=\"noopener noreferrer\">Web Scraping RPA<\/a> (Data Extraction)<\/strong><\/p><\/blockquote>\n<p><strong>Note:<\/strong> You can use several supported engines here, but if we use mechanize no configurations or installations are involved and work for simple HTTP requests but no javascript but if we use other engines such as selenium_chrome, poltergeist_phantomjs, selenium_firefox are all javascript based and rendered in <strong>HEADLESS<\/strong> mode.<\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>Add the parse method to the model for initiating the scrap process.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><script src=\"https:\/\/gist.github.com\/HunnyJummani\/458bfbc1968dd61161e85494211ddaa4.js\"><\/script><\/p>\n<p style=\"padding-left: 80px;\">Here, in the above parse method,<\/p>\n<p style=\"padding-left: 80px;\"><code>response<\/code> = Nokogiri::HTML::Document object for the requested website.<\/p>\n<p style=\"padding-left: 80px;\"><code>URL<\/code> = String URL of a processed web page.<\/p>\n<p style=\"padding-left: 80px;\"><code>data<\/code> = used to pass data between 2 requests.<\/p>\n<p>The data to be fetched from a website is selected using XPath and structures the data as per the requirement.<\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>Open the terminal and run the application using:<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p style=\"padding-left: 80px;\"><code>rails s<\/code><\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>Click on the link <code>'Start Scrap'<\/code>\n<ul>\n<li>The results will be saved in the <code>results.json<\/code> file using <code>save_to<\/code> helper of the gem.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>Now, check out the stored <code>JSON<\/code> file, you will get the scraped data.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>Hooray !! You have extracted information from the static website.<\/p>\n<ul>\n<li>Making a simple <a href=\"https:\/\/www.botreetechnologies.com\/blog\/securely-send-large-files-for-free-using-ruby-with-firefox-send\/\" target=\"_blank\" rel=\"noopener noreferrer\">ruby file<\/a> for extracting the information.<\/li>\n<\/ul>\n<ol>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>Open the terminal and install kimurai using the below-mentioned command:<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<p style=\"padding-left: 80px;\"><code>gem install kimurai<\/code><\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>You can refer to the code written for the generated model and make a ruby file using it.<\/li>\n<li>Run that ruby file using:<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p style=\"padding-left: 80px;\"><code>ruby filename.rb<\/code><\/p>\n<p><strong>Dynamic Websites \/ JS rendered websites:<\/strong><\/p>\n<p><strong>Pre-requisites:<\/strong><\/p>\n<p>Install browsers with web drivers:<\/p>\n<p><strong>For Ubuntu 18.04:<\/strong><\/p>\n<ul>\n<li>For automatic installation, use the <code>setup<\/code> command:<\/li>\n<\/ul>\n<p style=\"padding-left: 40px;\"><code>$ kimurai setup localhost --local --ask-sudo<\/code><\/p>\n<p style=\"padding-left: 40px;\"><strong>Note:<\/strong> It works using Ansible. If not installed, install using:<\/p>\n<p style=\"padding-left: 40px;\"><code>$ sudo apt install ansible<\/code><\/p>\n<ul>\n<li>Firstly, install basic tools:<\/li>\n<\/ul>\n<p style=\"padding-left: 40px;\"><code>sudo apt install -q -y unzip wget tar openssl<\/code><br \/>\n<code>sudo apt install -q -y xvfb<\/code><\/p>\n<ul>\n<li>For manual installation, follow the commands for the specific browsers.<\/li>\n<\/ul>\n<p><script src=\"https:\/\/gist.github.com\/HunnyJummani\/31b9a75bb4f2ea512acc28a26a6d252b.js\"><\/script><\/p>\n<p><strong>You can use this framework in 2 ways:<\/strong><\/p>\n<ul>\n<li>Making a rails app and extract information with the help of models and controllers.\n<ul>\n<li>Follow all the above steps from <code>a to o<\/code> for static websites.<\/li>\n<li>Change the @engine\u00a0 from <code>:mechanize<\/code> to <code>:selenium_chrome<\/code> \u00a0for using chrome driver for scraping.<\/li>\n<li>Also, change the parse method in the model to get the desired output.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><script src=\"https:\/\/gist.github.com\/HunnyJummani\/f5676d3c0bc1ac7e91e5ffc809bcf03e.js\"><\/script><\/p>\n<ul>\n<li>Making a simple ruby file for extracting the information.\n<ul>\n<li>Open the terminal and install kimurai using the below-mentioned command:<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p style=\"padding-left: 80px;\"><code>gem install kimurai<\/code><\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>You can refer to the code written for the generated model in the section of the dynamic website and make a ruby file using it.<\/li>\n<li>Run that ruby file using:<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p style=\"padding-left: 80px;\"><code>ruby filename.rb<\/code><\/p>\n<p>You can find the whole source code <a href=\"https:\/\/github.com\/HunnyJummani\/web_scraping_demo\" target=\"_blank\" rel=\"noopener noreferrer\">here<\/a>.<\/p>\n<p>Visit BoTree Technologies for excellent Ruby on Rails web development services and <a href=\"https:\/\/www.botreetechnologies.com\/hire-ruby-on-rails-developers\" target=\"_blank\" rel=\"noopener noreferrer\">hire Ruby on Rails web developers<\/a> with experience in handling marketplace development projects.<\/p>\n<p>Reach out to learn more about the New York web development agencies for the various ways to improve or build the quality of projects and across your company.<\/p>\n<p><strong><a href=\"https:\/\/www.botreetechnologies.com\/contact\">Consulting is free<\/a> &#8211; let us help you grow!<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"<p>What is Web Scraping? Web scraping is a technique to&#8230;<\/p>\n","protected":false},"author":56,"featured_media":13304,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[73,10],"tags":[],"class_list":["post-9175","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ruby-on-rails","category-technology"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.2 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Steps for web scraping with Ruby on Rails - BoTree Technologies<\/title>\n<meta name=\"description\" content=\"Web scraping with Ruby on Rails allows extracting data from websites in minimal time. Here are the steps for web scraping using Ruby on Rails.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Steps for web scraping with Ruby on Rails - BoTree Technologies\" \/>\n<meta property=\"og:description\" content=\"Web scraping with Ruby on Rails allows extracting data from websites in minimal time. Here are the steps for web scraping using Ruby on Rails.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/\" \/>\n<meta property=\"og:site_name\" content=\"BoTree Technologies\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/BoTreeTechnologies\/\" \/>\n<meta property=\"article:published_time\" content=\"2020-03-18T10:22:20+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-01-18T13:03:35+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.botreetechnologies.com\/blog\/wp-content\/uploads\/2020\/03\/web-scraping.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1024\" \/>\n\t<meta property=\"og:image:height\" content=\"683\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Hunny Jummani\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@BoTreeTech\" \/>\n<meta name=\"twitter:site\" content=\"@BoTreeTech\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Hunny Jummani\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/\"},\"author\":{\"name\":\"Hunny Jummani\",\"@id\":\"https:\/\/www.botreetechnologies.com\/blog\/#\/schema\/person\/0d616df8a58c8145fae7ceb8d048e23e\"},\"headline\":\"Steps for web scraping with Ruby on Rails\",\"datePublished\":\"2020-03-18T10:22:20+00:00\",\"dateModified\":\"2024-01-18T13:03:35+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/\"},\"wordCount\":1006,\"commentCount\":13,\"image\":{\"@id\":\"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.botreetechnologies.com\/blog\/wp-content\/uploads\/2020\/03\/web-scraping.jpg\",\"articleSection\":[\"Ruby on Rails\",\"Technology\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/\",\"url\":\"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/\",\"name\":\"Steps for web scraping with Ruby on Rails - BoTree Technologies\",\"isPartOf\":{\"@id\":\"https:\/\/www.botreetechnologies.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.botreetechnologies.com\/blog\/wp-content\/uploads\/2020\/03\/web-scraping.jpg\",\"datePublished\":\"2020-03-18T10:22:20+00:00\",\"dateModified\":\"2024-01-18T13:03:35+00:00\",\"author\":{\"@id\":\"https:\/\/www.botreetechnologies.com\/blog\/#\/schema\/person\/0d616df8a58c8145fae7ceb8d048e23e\"},\"description\":\"Web scraping with Ruby on Rails allows extracting data from websites in minimal time. Here are the steps for web scraping using Ruby on Rails.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/#primaryimage\",\"url\":\"https:\/\/www.botreetechnologies.com\/blog\/wp-content\/uploads\/2020\/03\/web-scraping.jpg\",\"contentUrl\":\"https:\/\/www.botreetechnologies.com\/blog\/wp-content\/uploads\/2020\/03\/web-scraping.jpg\",\"width\":1024,\"height\":683,\"caption\":\"web scraping ruby on rails\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.botreetechnologies.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Steps for web scraping with Ruby on Rails\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.botreetechnologies.com\/blog\/#website\",\"url\":\"https:\/\/www.botreetechnologies.com\/blog\/\",\"name\":\"BoTree Technologies\",\"description\":\"Committed to inspire generation.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.botreetechnologies.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.botreetechnologies.com\/blog\/#\/schema\/person\/0d616df8a58c8145fae7ceb8d048e23e\",\"name\":\"Hunny Jummani\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.botreetechnologies.com\/blog\/wp-content\/uploads\/2020\/10\/hunny-jummani-150x150.png\",\"url\":\"https:\/\/www.botreetechnologies.com\/blog\/wp-content\/uploads\/2020\/10\/hunny-jummani-150x150.png\",\"contentUrl\":\"https:\/\/www.botreetechnologies.com\/blog\/wp-content\/uploads\/2020\/10\/hunny-jummani-150x150.png\",\"caption\":\"Hunny Jummani\"},\"description\":\"I am a Ruby On Rails Developer. I love JavaScript, logical problem solving in Ruby and exploration of new things. Apart from coding, I also like to travel, listen music and play guitar.\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Steps for web scraping with Ruby on Rails - BoTree Technologies","description":"Web scraping with Ruby on Rails allows extracting data from websites in minimal time. Here are the steps for web scraping using Ruby on Rails.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/","og_locale":"en_US","og_type":"article","og_title":"Steps for web scraping with Ruby on Rails - BoTree Technologies","og_description":"Web scraping with Ruby on Rails allows extracting data from websites in minimal time. Here are the steps for web scraping using Ruby on Rails.","og_url":"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/","og_site_name":"BoTree Technologies","article_publisher":"https:\/\/www.facebook.com\/BoTreeTechnologies\/","article_published_time":"2020-03-18T10:22:20+00:00","article_modified_time":"2024-01-18T13:03:35+00:00","og_image":[{"width":1024,"height":683,"url":"https:\/\/www.botreetechnologies.com\/blog\/wp-content\/uploads\/2020\/03\/web-scraping.jpg","type":"image\/jpeg"}],"author":"Hunny Jummani","twitter_card":"summary_large_image","twitter_creator":"@BoTreeTech","twitter_site":"@BoTreeTech","twitter_misc":{"Written by":"Hunny Jummani","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/#article","isPartOf":{"@id":"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/"},"author":{"name":"Hunny Jummani","@id":"https:\/\/www.botreetechnologies.com\/blog\/#\/schema\/person\/0d616df8a58c8145fae7ceb8d048e23e"},"headline":"Steps for web scraping with Ruby on Rails","datePublished":"2020-03-18T10:22:20+00:00","dateModified":"2024-01-18T13:03:35+00:00","mainEntityOfPage":{"@id":"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/"},"wordCount":1006,"commentCount":13,"image":{"@id":"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/#primaryimage"},"thumbnailUrl":"https:\/\/www.botreetechnologies.com\/blog\/wp-content\/uploads\/2020\/03\/web-scraping.jpg","articleSection":["Ruby on Rails","Technology"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/","url":"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/","name":"Steps for web scraping with Ruby on Rails - BoTree Technologies","isPartOf":{"@id":"https:\/\/www.botreetechnologies.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/#primaryimage"},"image":{"@id":"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/#primaryimage"},"thumbnailUrl":"https:\/\/www.botreetechnologies.com\/blog\/wp-content\/uploads\/2020\/03\/web-scraping.jpg","datePublished":"2020-03-18T10:22:20+00:00","dateModified":"2024-01-18T13:03:35+00:00","author":{"@id":"https:\/\/www.botreetechnologies.com\/blog\/#\/schema\/person\/0d616df8a58c8145fae7ceb8d048e23e"},"description":"Web scraping with Ruby on Rails allows extracting data from websites in minimal time. Here are the steps for web scraping using Ruby on Rails.","breadcrumb":{"@id":"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/#primaryimage","url":"https:\/\/www.botreetechnologies.com\/blog\/wp-content\/uploads\/2020\/03\/web-scraping.jpg","contentUrl":"https:\/\/www.botreetechnologies.com\/blog\/wp-content\/uploads\/2020\/03\/web-scraping.jpg","width":1024,"height":683,"caption":"web scraping ruby on rails"},{"@type":"BreadcrumbList","@id":"https:\/\/www.botreetechnologies.com\/blog\/how-to-do-web-scraping-of-a-static-or-dynamic-website-with-ruby-on-rails\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.botreetechnologies.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Steps for web scraping with Ruby on Rails"}]},{"@type":"WebSite","@id":"https:\/\/www.botreetechnologies.com\/blog\/#website","url":"https:\/\/www.botreetechnologies.com\/blog\/","name":"BoTree Technologies","description":"Committed to inspire generation.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.botreetechnologies.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.botreetechnologies.com\/blog\/#\/schema\/person\/0d616df8a58c8145fae7ceb8d048e23e","name":"Hunny Jummani","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.botreetechnologies.com\/blog\/wp-content\/uploads\/2020\/10\/hunny-jummani-150x150.png","url":"https:\/\/www.botreetechnologies.com\/blog\/wp-content\/uploads\/2020\/10\/hunny-jummani-150x150.png","contentUrl":"https:\/\/www.botreetechnologies.com\/blog\/wp-content\/uploads\/2020\/10\/hunny-jummani-150x150.png","caption":"Hunny Jummani"},"description":"I am a Ruby On Rails Developer. I love JavaScript, logical problem solving in Ruby and exploration of new things. Apart from coding, I also like to travel, listen music and play guitar."}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/www.botreetechnologies.com\/blog\/wp-json\/wp\/v2\/posts\/9175","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.botreetechnologies.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.botreetechnologies.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.botreetechnologies.com\/blog\/wp-json\/wp\/v2\/users\/56"}],"replies":[{"embeddable":true,"href":"https:\/\/www.botreetechnologies.com\/blog\/wp-json\/wp\/v2\/comments?post=9175"}],"version-history":[{"count":4,"href":"https:\/\/www.botreetechnologies.com\/blog\/wp-json\/wp\/v2\/posts\/9175\/revisions"}],"predecessor-version":[{"id":19787,"href":"https:\/\/www.botreetechnologies.com\/blog\/wp-json\/wp\/v2\/posts\/9175\/revisions\/19787"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.botreetechnologies.com\/blog\/wp-json\/wp\/v2\/media\/13304"}],"wp:attachment":[{"href":"https:\/\/www.botreetechnologies.com\/blog\/wp-json\/wp\/v2\/media?parent=9175"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.botreetechnologies.com\/blog\/wp-json\/wp\/v2\/categories?post=9175"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.botreetechnologies.com\/blog\/wp-json\/wp\/v2\/tags?post=9175"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}