Objective:

The scrapper built as a wordpres plugin so the scrapped data will be managed through wordpress admin as a custom post type as well as the few pages created to manage and displaying the processing information of the URLs.

Strategy:

The program scraping all the movies and Tv shows (Sessions and Episodes) and stored in the wordpress posts under custom post types “Movies” and “Tv Shows”. All the additional information are saved in post Meta options

Scrapped Movies:

Scrapped TV Shows:

Advance Features:

Short-codes
Plugin supports displaying tv shows and movies using short codes and comes with two short codes.

Placing these two short codes in any page will convert it and display all the scrapped movies or TV shows with pagination. (Displaying 30 items per page)

Settings
The settings panel support for proxies and other general settings for scrapping and displaying data in wordpress pages. And upon installation of the plugin it will create movies and tv shows list display pages automatically and enables to change from settings in any case.

CronJobs
As the complexity and the quantity of data the program is built to work with 3 cron jobs to complete the movie scrape and stored process and the for Tv shows it is 4 cron jobs (including processing sessions and episodes for each session)

Data Usage
All the scrapped data are stored in wordpress database and deliver as posts to site visitors. The Meta options used to store additional details including movie watch links and etc.

Manage Random proxies for scrapping
The plugin support proxies in its settings panel and loads the proxes randomly in a way to make sure that the ips are won’t get blacklist.

Multi Treaded processing
Processor can scrape nearly 10 movies and 1 TV show with 10 Episodes every minute. Due to complexity of data the processing is slower but still support the multi treaded for faster processing