The WWW::Mechanize Perl module is used to interact with web pages the way a browser might.

Perl CPAN site is here: http://search.cpan.org/dist/WWW-Mechanize/lib/WWW/Mechanize/Cookbook.pod

Installation

Installing Mechanize

I first installed cpanm (following instructions on the Perl page).

Then I had lots of problems with Mechanize's dependencies failing tests. I think it's a lot easier to install various Perl modules if you run it as the superuser, which I wasn't doing.

I found a site [1] where someone gave a nice script to take care of all that stuff (note that it must be run as sudo):

#!/usr/bin/perl

# from:
# http://www.perlmonks.org/?node_id=901715

use strict;
use warnings;
use CPAN;

CPAN::Shell->install(
"Tree::DAG_Node",
"Test::Warn",
"WWW::RobotRules",
"HTTP::Cookies",
"Net::FTP",
"Net::HTTP",
"Digest::base",
"Digest::MD5",
"HTTP::Negotiate",
"File::Listing",
"LWP::UserAgent",
"HTML::Form",
"HTTP::Daemon",
"Test",
"Text::Wrap",
"Pod::Escapes",
"Pod::Simple",
"Pod::Man",
"ExtUtils::MakeMaker",
"HTTP::Server::Simple",
"Time::Local",
"HTTP::Date",
"MIME::Base64",
"URI",
"Encode",
"Encode::Locale",
"LWP::MediaTypes",
"Compress::Raw::Bzip2",
"Compress::Raw::Zlib",
"IO::Uncompress::Inflate",
"HTTP::Status",
"ExtUtils::ParseXS",
"Module::CoreList",
"Module::Load",
"Params::Check",
"Module::Load::Conditional",
"Locale::Maketext::Simple",
"IPC::Cmd",
"ExtUtils::CBuilder",
"Perl::OSType",
"IO::Dir",
"Version::Requirements",
"Exporter",
"CPAN::Meta::YAML",
"JSON::PP",
"Parse::CPAN::Meta",
"CPAN::Meta",
"Scalar::Util",
"File::Spec",
"File::Temp",
"version",
"Module::Metadata",
"Module::Build",
"HTML::Tagset",
"XSLoader",
"HTML::Parser",
"Sub::Uplevel",
"Test::Exception",
"HTML::TreeBuilder",
"Test::Harness",
"Test::More",
"Pod::Usage",
"WWW::Mechanize");

References

General Mechanize Links

CPAN Perl Mechanize page:

http://search.cpan.org/~jesse/WWW-Mechanize/

Introduction to screen scraping with Perl:

http://www.perl.com/pub/2003/01/22/mechanize.html

Cookies

HTTP Cookies:

http://search.cpan.org/~gaas/HTTP-Cookies-6.00/lib/HTTP/Cookies.pm

HTTP Cookies Mozilla:

http://search.cpan.org/~polettix/HTTP-Cookies-Mozilla-2.03/lib/Mozilla.pm

Processing HTML with Perl

HTML::TokeParser to parse HTML tokens/tags:

HTML::Entities (specifically, decode_entities):

http://www.xav.com/perl/site/lib/HTML/Entities.html

HTTPS with Mechanize

Accessing secure login sites (HTTPS) with Perl:

http://www.ibm.com/developerworks/linux/library/wa-perlsecure/index.html

Mechanize (Perl)

From charlesreid1

Contents