Objective:
This scrapper built as a wordpress plugin and for scraping the eBook information from Frasesbonitas.biz and store in a wordpress custom post type called “EBooks”. The eBook categories are stored in wordpress taxonomy called “Ebook_Categories” and all the eBooks are stored in their original category as in the first site.
Strategy:
Plugin itself has few pages for manage and displaying the processing information of the eBooks. All the additional information about eBooks are saved in post Meta options of each post.
Scrapped Books:
Following data scrapped for ebooks from the main site and stored as WordPress posts.
- Ebook Title
- Cover image
- Description
- Reviews
- About Author
- Author URL
- Publisher
- Publisher URL
- ISBN
Advance Features:
Short-codes
Plugin supports displaying eBooks using short codes and comes with two short codes.
1. [fbbs_list_category_menu] – Display categories menu
2. [fbbs_all_ebooks_or_by_category] – List all posts from a category
Settings
The settings panel support for proxies and other general settings for scrapping and displaying data in wordpress pages. And upon installation of the plugin it will create two pages for display all ebooks (archive) and the single ebook view page which can change from settings panel later.
CronJobs
As the complexity and the quantity of data the program is built to work with 4 cron jobs to complete the ebook scrape and stored process
Data Usage
All the scrapped data are stored in wordpress database and deliver as posts to site visitors. The Meta options used to store additional details including movie watch links and etc.
Manage Random proxies for scrapping
The plugin support proxies in its settings panel and loads the proxes randomly in a way to make sure that the ips are won’t get blacklist.
Multi Treaded processing
Processor can scrape nearly 10 eBooks in every minute. Due to complexity of data the processing and as the data are gathered through different pages for a eBook, it is slower but still support the multi treaded for faster processing
