Re: [IBMiOSS] Latest node installation woes -- OPENSOURCE

Job creation time on IBM i is very expensive compared to say Linux or
other Unix operating systems, since these operating systems are designed
around that execution model. IBM i was not designed around forking, but
more of a spawn style execution model. Calling IBM i ILE and OPM programs
is more akin to calling a function in Linux/Unix.

For the vast majority of PASE applications changing the ratio of PASE
process to IBM i job from 1:1 to N:1 would be more ideal, but that would
require quite extensive changes in SLIC for PASE as well as Work
Management and such so I'm not sure how much traction I'd get.

"OpenSource" <opensource-bounces@xxxxxxxxxxxx> wrote on 10/04/2016
12:49:30 PM:

From: Aaron Bartell <aaronbartell@xxxxxxxxx>
To: IBMi Open Source Roundtable <opensource@xxxxxxxxxxxx>
Date: 10/04/2016 12:51 PM
Subject: Re: [IBMiOSS] Latest node installation woes
Sent by: "OpenSource" <opensource-bounces@xxxxxxxxxxxx>

Wowza! Nice response! I've documented it here for easy

finding/modifying:

https://bitbucket.org/ibmi/opensource/wiki/PASE. If you'd prefer to do

article instead then feel free to delete that. I am favoring wikis over
articles because articles can't be edited.

If you pay close enough attention above, you'll see that each of these

programs run in a new process (job), so there is no way a cached version

the process could have affected it.

Is there documentation of what, if anything, is inherited from the

parent

process? For example, environment variables.
http://unix.stackexchange.com/questions/17758/exception-of-
inheritance-of-environment-variables

You mention a lot of new IBM i jobs being created. Do you know how much
more "expensive" forks are, CPU wise, as compared to Linux? I ask

because

I watch things like chroot environment creation and it takes decently

long

and creates a fair amount of jobs (non-scientific watching of WRKACTJOB)
that pop in and out of existence (as you've described). Maybe for PASE
things we need "fast fork"(n1)? Or do we already have something like

that?

n1 -
http://www.ibm.com/support/knowledgecenter/ssw_aix_61/
com.ibm.aix.performance/proc_threads.htm

Aaron Bartell
litmis.com - Services for open source on IBM i

On Tue, Oct 4, 2016 at 11:54 AM, Kevin Adler <kadler@xxxxxxxxxx> wrote:

At program startup the linker/loader will analyze all the dependent
libraries of the program and load them in to the program's address

space.

Any dependencies of those libraries will also be loaded as well (I

don't

know if it's done depth-first or breadth-first). Then static
initialization occurs and finally the program entry point (main) is
called. These libraries are loaded in to your address space until

program

termination or you call exec(). You can dynamically load at runtime
through dlopen() API set and in that case, libraries can become

unloaded

(though it's not recommended as many libraries do not handle this
properly).

What you're asking about is completely different, though. Let's say

you're

at a shell prompt and run some perzl application:

$ /opt/freeware/bin/curl http://google.com

- The shell will first fork a new child process
- In the child, it will pass the program and its arguments to exec()

(or

some variant thereof).
- exec will overwrite the process's address space with the program

binary

- the loader will load all the dependent libraries as detailed above,
including /opt/freeware/lib/libcurl.a which then causes
/opt/freeware/lib/libssl.a to be loaded
- curl downloads the url and echoes it to the screen
- the parent (shell) is waiting for the child to exit and finally gets
signalled that it has
- the parent reads the child's exit code and sets $?

Now you call node:

$ /QOpenSys/QIBM/ProdData/Node/bin/node -v

- The shell forks a new child process
- In the child, again passes node and the arguments to exec
- exec overwrites the child's address space with the node binary
- the loader will load all of node's dependent libraries, including
/QOpenSys/usr/lib/libssl.a
- node prints out its version and exits with a 0 return code
- the parent gets signalled that the child has exited
- the parent reads the child's exit code and sets $?

If you pay close enough attention above, you'll see that each of these
programs run in a new process (job), so there is no way a cached

version

in the process could have affected it. When you do an exec, everything

the process is replaced by a new program so you don't have to worry

about

that.

Now, AIX does have the concept of a system-wide cache. Basically when

library is first loaded, it will be loaded in to this system wide

cache

and mapped in to memory. Then instead of having to load from disk for
every process that wants to use the library, the linker just has to

map

those pages in to the process's address space. You can run slibclean

clear this cache of any libraries that are not in use. This cache is
affected by loader domains as well as the permissions set on the

library

itself. If a library is set executable only by the owner, it is loaded

to a private mapping and is not cached.

The other thing to remember is that the library is really just an

archive

of shared objects (.so or .o) or static objects (.o).

For instance, looking at libcurl.a, we see it depends on libssl.a, but
really the libssl.so.0.9.8 shared object inside the libssl.a archive:
/opt/freeware/lib/libcurl.a[libcurl.so.4]:

***Loader Section***
Loader Header Information
VERSION# #SYMtableENT #RELOCent LENidSTR
0x00000001 0x000002b9 0x00000aef 0x00000102

#IMPfilID OFFidSTR LENstrTBL OFFstrTBL
0x00000009 0x0000c4ac 0x00002f31 0x0000c5ae

***Import File Strings***
INDEX PATH BASE MEMBER
0 /opt/freeware/lib:/opt/freeware/lib:/usr/vac/lib:/usr/lib:/lib

1 libc.a shr.o
2 libidn.a libidn.so.11
3 libcrypto.a libcrypto.so.0.9.8
4 libssl.a libssl.so.0.9.8
5 libz.a libz.so.1
6 libldap.a libldap-2.4.so.2

7 liblber.a liblber-2.4.so.2

8 libssh2.a libssh2.so.1

Also looking at index 0 is where you find the compiled in LIBPATH for

this

library. This is where the linker will look for these dependent

libraries

when loading libcurl.a.

When looking at node, however, we see that it is dependent on

libssl.so.1

shared object in libssl.a:

kadler@wernstrom:~>dump -H /QOpenSys/QIBM/ProdData/OPS/Node4/bin/node

/QOpenSys/QIBM/ProdData/OPS/Node4/bin/node:

***Loader Section***
Loader Header Information
VERSION# #SYMtableENT #RELOCent LENidSTR
0x00000001 0x00005904 0x00013a47 0x000001da

#IMPfilID OFFidSTR LENstrTBL OFFstrTBL
0x00000006 0x001713d4 0x00173f11 0x001715ae

***Import File Strings***
INDEX PATH BASE MEMBER
0
.:/QOpenSys/opt/freeware/bin/../lib/gcc/powerpc-ibm-aix6.1.
0.0/4.8.4/pthread:/QOpenSys/opt/freeware/bin/../lib/gcc/
powerpc-ibm-aix6.1.0.0/4.8.4/../../../pthread:/QOpenSys/opt/
freeware/bin/../lib/gcc/powerpc-ibm-aix6.1.0.0/4.8.4:/
QOpenSys/opt/freeware/bin/../lib/gcc:/QOpenSys/opt/
freeware/bin/../lib/gcc/powerpc-ibm-aix6.1.0.0/4.8.4/.
./../..:/usr/lib:/lib

1 libc.a shr.o
2 libpthreads.a shr_xpg5.o
3 libcrypto.a libcrypto.so.1
4 libpthreads.a shr_comm.o
5 libssl.a libssl.so.1

It has quite a complicated LIBPATH set, since by default the linker

will

add any path you specify with -L to this LIBPATH and GCC adds a bunch

its own paths automatically. So the only time you would run in to a
problem is if you had a libssl.a that was in one of these common paths
between the two binaries that did not contain both dependent objects
(libssl.so.0.9.8 and libssl.so.1).

Anyway, quite a rambling knowledge dump. I should probably polish it a

bit

and turn it in to an article or something...

"OpenSource" <opensource-bounces@xxxxxxxxxxxx> wrote on 10/03/2016
07:43:30 PM:

From: Aaron Bartell <aaronbartell@xxxxxxxxx>
To: IBMi Open Source Roundtable <opensource@xxxxxxxxxxxx>
Date: 10/03/2016 07:43 PM
Subject: Re: [IBMiOSS] Latest node installation woes
Sent by: "OpenSource" <opensource-bounces@xxxxxxxxxxxx>

Ignorance question: when libxxxxx.a resolution happens is there ever

cached version in the job's memory based on it being previously

loaded?

That's where my PATH comments are coming from. Specifically, if

perzl

stuff

was run immediately previous to "node -v" could it effect it if both

were

going after libcrypto.a?

On Oct 3, 2016 5:49 PM, "Kevin Adler" <kadler@xxxxxxxxxx> wrote:

"OpenSource" <opensource-bounces@xxxxxxxxxxxx> wrote on 10/03/2016
07:02:06 AM:

From: Aaron Bartell <aaronbartell@xxxxxxxxx>
To: IBMi Open Source Roundtable <opensource@xxxxxxxxxxxx>
Date: 10/03/2016 07:02 AM
Subject: Re: [IBMiOSS] Latest node installation woes
Sent by: "OpenSource" <opensource-bounces@xxxxxxxxxxxx>

Further to Robert's comments, you should also run the
/QOpenSys/QIBM/ProdData/OPS/Node4/bin/nodeenv.sh(n1) shell

script

which

will set the symlinks in /QOpenSys/usr/bin.

n1 - More info here: http://bit.ly/developerWorks-node-v4

Concerning why it bombs on libcrypto.a, that many times is

because

of a

conflict with perzl stuff. Remove /opt/freeware/bin from PATH

and

see

if

that allows it to continue. If you need perzl stuff in your

path

then

that's a longer conversation to fix it.

PATH doesn't matter for finding libraries (only binaries). Only

LIBPATH

would matter in this case (don't set LIBPATH unless you know what

you're

doing).

Also, it appears you're using Node
v.0.12.x(/QOpenSys/QIBM/ProdData/Node/bin/node). IBM has

alluded

giving

that an EndOfLife, time to move on to Node
v4.x(/QOpenSys/QIBM/ProdData/OPS/Node4/bin/node).

Aaron Bartell
litmis.com - Services for open source on IBM i

On Mon, Oct 3, 2016 at 5:11 AM, R Brown <rbpaservices@xxxxxxxxx>

wrote:

I had the same issue. Look inside the OPS for NODE4 and add

that

to

the

path. It should find node then.

On Oct 3, 2016 4:40 AM, "Kevin Turner"

<kevin.turner@xxxxxxxxxxxxxxx>

wrote:

Hi

The guys managing our US IBMi servers are now trying to

install

5770OPS.

They have installed the product and all PTFs, but it is

still

foobar.

First and foremost no symbolic links have been created in

/usr/bin

for

the

“node” and “npm” commands.
Despite that, even if you navigate to

QOpenSys/QIBM/ProdData/Node/bin

and

execute the node command it fails:

node -v
exec(): 0509-036 Cannot load program node because of the

following

errors:

0509-150 Dependent module

libcrypto.a(libcrypto.so.1)

could not

be loaded.
0509-022 Cannot load module

libcrypto.a(libcrypto.so.1).

0509-026 System error: A file or directory in the

path

name

does

not exist.

Sounds like something else needs to be there that 5770OPS

does

not

install

(ssh perhaps)?

I am not sure how or why IBM are making such a pigs ear of

this.

Anyone

seen this before?

Thanks
Kevin

________________________________

NOTICE: The information in this electronic mail transmission

intended

by CoralTree Systems Ltd for the use of the named

individuals or

entity

to

which it is directed and may contain information that is

privileged

or

otherwise confidential. If you have received this electronic

mail

transmission in error, please delete it from your system

without

copying

or

forwarding it, and notify the sender of the error by reply

or

by

telephone, so that the sender's address records can be

corrected.

------------------------------------------------------------
--------------------

CoralTree Systems Limited
25 Barnes Wallis Road
Segensworth East, Fareham
PO15 5TT

Company Registration Number 5021022.
Registered Office:
12-14 Carlton Place
Southampton, UK
SO15 2EA
VAT Registration Number 834 1020 74.
--
This is the IBMi Open Source Roundtable (OpenSource) mailing

list

To post a message email: OpenSource@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/opensource
or email: OpenSource-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/opensource.

--
This is the IBMi Open Source Roundtable (OpenSource) mailing list
To post a message email: OpenSource@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/opensource
or email: OpenSource-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/opensource.

--
This is the IBMi Open Source Roundtable (OpenSource) mailing list
To post a message email: OpenSource@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/opensource
or email: OpenSource-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/opensource.

--
This is the IBMi Open Source Roundtable (OpenSource) mailing list
To post a message email: OpenSource@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/opensource
or email: OpenSource-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/opensource.

--
This is the IBMi Open Source Roundtable (OpenSource) mailing list
To post a message email: OpenSource@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/opensource
or email: OpenSource-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/opensource.