Re: [IBMiOSS] Headless Browsing -- OPENSOURCE

You can update pip, if you like. When it updates, it will install to a
different location than where the PTF installs pip so it won't mess up PTF
upgrades. To upgrade pip3, you would need to run pip3 install --upgrade
pip

If you just want to get rid of the warning message, you can disable it by
adding the following to your pip.conf:

[global]
disable_pip_version_check = 1

You can create a per-use pip.conf here:
$HOME/.pip/pip.conf

or a system-level pip.conf here:
/etc/pip.conf

----- Original message -----
From: Roger Harman <roger.harman@xxxxxxxxxxx>
Sent by: "OpenSource" <opensource-bounces@xxxxxxxxxxxx>
To: IBMi Open Source Roundtable <opensource@xxxxxxxxxxxx>
Cc:
Subject: Re: [IBMiOSS] Headless Browsing
Date: Mon, Oct 2, 2017 7:02 PM

You nailed it when you said "examples, tutorials....". I, too, think
that Kevin's suggestion of MechanicalSoup bears looking into - I
completely forgot about John's mention of it.

Git still confounds me but I see MechanicalSoup is installable via pip
and I've done so.

On another note... every time I run pip (actually, pip3), I get this:
You are using pip version 8.1.1, however version 9.0.1 is
available.
You should consider upgrading via the 'pip install --upgrade
pip' command.

I've tried various incantations of that suggestion to no avail. Is pip
(pip3), on the i, upgradeable in this fashion or is it via PTF?

Aaaargghh... RoboBrowser - Another option to consider! Jeez, I wish the
site I'm using had a web service.

Roger Harman
COMMON Certified Application Developer - ILE RPG on IBM i on Power

From: OpenSource <opensource-bounces@xxxxxxxxxxxx> on behalf of John
Yeung <gallium.arsenide@xxxxxxxxx>
Sent: Monday, October 2, 2017 3:34 PM
To: IBMi Open Source Roundtable
Subject: Re: [IBMiOSS] Headless Browsing

On Mon, Oct 2, 2017 at 4:58 PM, Pete Helgren <pete@xxxxxxxxxx> wrote:
> Since we are all straying outside the Python fold, I use JSoup to
accomplish
> something similar in Java.....similar I guess to "Beautiful Soup"...

Well, the purpose of straying outside Python has mainly been to say
"here's something that worked for me; you can probably try the
equivalent thing in Python". Jack's response was a little more
mysterious; he just tossed in a link for Selenium, which OP already
mentioned he had looked into.

You are correct that jsoup is pretty much Java's BeautifulSoup, but
BeautifulSoup came first (by about 5 years, and very likely was a
strong influence on the development of jsoup), and it was the other
thing OP already mentioned besides Selenium.

So far Kevin's response has probably been the most helpful. There's no
way he could have known I already recommended MechanicalSoup to OP a
few months ago, when he approached me about this topic off-list.

I think what Roger is mainly missing right now is examples, tutorials,
and specific tips that come from experience, particularly from those
who are familiar with any IBM-i-specific wrinkles in the endeavor.

Unfortunately, I only have a tiny bit of experience with Selenium, via
Python on Windows. I never got to the point where I felt like I really
knew what I was doing (and a few months later, when I had some time to
try to expand on my early experiments, I discovered the site I was
trying to interact with changed some stuff which broke what I'd done
earlier). But a lot of that is just me not knowing much about Web
stuff in general.

I will say I agree with Kevin that if you can achieve what you need
without a full-blown browser (which is what Selenium is), it's usually
preferable to do so. Of course, you can't always tell what you can
achieve with something until you try it, so there's not really much
for it except to try it. I would try MechanicalSoup or RoboBrowser
first. (Whichever one seems friendlier or more intuitive.)

Roger, have you already attempted anything? If so, where are you getting
stuck?

John Y.
--
This is the IBMi Open Source Roundtable (OpenSource) mailing list
To post a message email: OpenSource@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit:
[1]https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.midrange.com_mailman_listinfo_opensource&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=YPtV4_SMxJTLXtcvDJsBFyGeqQS_kDdaT_eQt56K40Y&s=EcGLVEZ9fFKh11EYNVp-DjQxRv8r66GxUq9vNqDS3gM&e=
or email: OpenSource-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at
[2]https://urldefense.proofpoint.com/v2/url?u=https-3A__archive.midrange.com_opensource&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=YPtV4_SMxJTLXtcvDJsBFyGeqQS_kDdaT_eQt56K40Y&s=cun4cmzUG7km3x0YdNs32HHMLhig4wF3PdqHyuC3_po&e= .

--
This is the IBMi Open Source Roundtable (OpenSource) mailing list
To post a message email: OpenSource@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit:
[3]https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.midrange.com_mailman_listinfo_opensource&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=YPtV4_SMxJTLXtcvDJsBFyGeqQS_kDdaT_eQt56K40Y&s=EcGLVEZ9fFKh11EYNVp-DjQxRv8r66GxUq9vNqDS3gM&e=
or email: OpenSource-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at
[4]https://urldefense.proofpoint.com/v2/url?u=https-3A__archive.midrange.com_opensource&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=YPtV4_SMxJTLXtcvDJsBFyGeqQS_kDdaT_eQt56K40Y&s=cun4cmzUG7km3x0YdNs32HHMLhig4wF3PdqHyuC3_po&e= .

References

Visible links
1. https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.midrange.com_mailman_listinfo_opensource&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=YPtV4_SMxJTLXtcvDJsBFyGeqQS_kDdaT_eQt56K40Y&s=EcGLVEZ9fFKh11EYNVp-DjQxRv8r66GxUq9vNqDS3gM&e=
2. https://urldefense.proofpoint.com/v2/url?u=https-3A__archive.midrange.com_opensource&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=YPtV4_SMxJTLXtcvDJsBFyGeqQS_kDdaT_eQt56K40Y&s=cun4cmzUG7km3x0YdNs32HHMLhig4wF3PdqHyuC3_po&e=
3. https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.midrange.com_mailman_listinfo_opensource&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=YPtV4_SMxJTLXtcvDJsBFyGeqQS_kDdaT_eQt56K40Y&s=EcGLVEZ9fFKh11EYNVp-DjQxRv8r66GxUq9vNqDS3gM&e=
4. https://urldefense.proofpoint.com/v2/url?u=https-3A__archive.midrange.com_opensource&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=YPtV4_SMxJTLXtcvDJsBFyGeqQS_kDdaT_eQt56K40Y&s=cun4cmzUG7km3x0YdNs32HHMLhig4wF3PdqHyuC3_po&e=