RE: Multi Occurrence Data Structure returned from a procedure

>> Passing a pointer as a "both" parameter (meaning both input and output) will 
>> work fine. However, don't RETURN
>>a pointer unless you understand the ramifications of doing so, and have taken 
>>appropriate precautions. 

Returning pointers from a service program or external routine is extremely bad 
form no matter what. The only time that I find it appropriate is where it must 
be returned as with a USPC_GetPointerToUserSpace.

I am, also, opposed to passing pointers to service programs. It is just too 
dangerous a practice as the receiving program has no idea what the hell it is 
receiving. Could be a pointer to anything and I have found little reason to do 
so. Passing a variable by reference does the same thing except the operating 
system is making sure what you pass is valid. 

The following is a write-up I did for your office on returning pointers and the 
problems I see with them. This is kind of long. Sorry.  

====================================================================================================================

Recently it has come to my attention that we have been returning pointers in 
RPG/ILE from service programs to return multiple records. I am not only 
concerned because this as happened in the past but we are talking about 
creating further service programs doing this. 
 
I have a series of objections to the practice and feel it is very poor 
programming practice to return a pointer from a service program. 
 
The only exception to that rule that I know of is a service program that 
returns a pointer to a user space. 
 
I have been writing service programs for eight years and many times have been 
faced with returning multiple records to caller. Many times I have considered 
using pointers but have always rejected them for the reasons I state below. I 
have never found a need to use a pointer in order to get multiple records or 
information from a service program. 
 
My concerns are on several levels.

Number 1 - Violation of Software Engineering Principals 
 
Returning a pointer to a user program violates the number one rule of Software 
Engineering, Encapsulation or Information Hiding as you might want to call it. 
 
You never expose details of how a function is implemented to the user of a 
service. The internal implementation of a list of items might be a fixed array, 
a dynamic array, a linked list, a pointer to a user space or even a cursor from 
an SQL statement. 
 
Returning a pointer says how you have implemented the function. The number one 
problem is doing this is you are now restricted if you want to change the 
function. If you suddenly realize in the future that you need to re-implement 
using a linked list or even an SQL cursor, you are out of luck because a linked 
list or cursor requires program logic to read through a list. 
 
Having exposed the internals, you cannot change the internal data structures. 
If you touch the internal data structures, you break any program using it 
because anyone using the pointer is basing their processing on pointer math 
either directly or through a pointer to a multiple occurrence data structure 
and there is no way for the user of the function to know that the internal 
structures have changed. You will have to go back and redevelop every program 
that uses that service program. If you miss one, you have a real mess. 
 
In addition, you are violating the most basic rules concerning implementation 
of a procedure or function. 
 
A procedure or function should perform one function and only one function and 
you should be able to describe what the function does without the use of 
conjunctive verbs like AND or OR. There should be a known set of inputs and a 
known set of outputs
 
In other words, functional decomposition. Breaking the problem down into 
smaller and smaller pieces until each piece is responsible for doing one thing.

If a function returns a pointer, you are saying something like 
GetListofStylesAndReturnList instead of something like BuildStyleList and 
GetStyleRecord.  

Number 2 - The consumer of the service must implement the logic.
 
Hardly less important is the fact that the user of the service program must 
implement the logic to retrieve the data instead of the programmer who wrote 
the service program. 
 
Pointers in the wrong hands can be very dangerous. The person using the pointer 
can do just about anything he or she wants to do with it. Making a mistake in 
implementation means walking all over memory in the service program. Results 
are completely unknown and there is no way to error check to see if what they 
are doing is correct. It is a simple hope and by god kind of thing. 
 
Under no circumstances should a consumer of a service have to implement the 
product that he or she is using but that is what we are asking the programmer 
to do. Instead of simply calling a function, they are forced to write 
implementation logic, i.e. pointer math to walk the list.  
 
A bigger issue arises in the fact the very few programmers on the AS/400 are 
familiar with or feel comfortable with pointers but by implementing the service 
program with a pointer, you are forcing a consumer of the service program to 
use and implement pointers no matter how comfortable they feel they are in 
using them.  

Number 3 - No way to validate function.
 
The reason that we build small functions is to be able to have a way to prove 
that a function works correctly. If we have a function with that does one thing 
with a known set of inputs and known outputs, we can test that function 
independently of the program we are working on. In other words, a black box. 
 
If a pointer is returned to a consumer, how do you verify that the function 
works correctly? You do not have any known set of outputs. What comes out is 
completely dependent on the person who implements the logic. Depending on how 
they do it, you will get a different output. The function is un-provable. 

Result of implementation methods.
 
I would like to offer an example of how the same problem of doing disk I/O was 
implemented in two systems.
In discussing alternative to using pointers, I think we must first discuss how 
languages like ILE/RPG are written and implemented. 
 
The whole concept in creating languages like ILE/RPG is to be able to break 
problems down into small pieces and provide for reusability. The reasons for 
wanting to do this are obvious if you have ever tried to debug a 10,000 line 
monolith program and tried to change a function that was implemented 
differently in a dozen programs. 
 
We want to be able to write and implement logic in one place. 
 
When a problem is broken down into small pieces, you have many small functions 
that are called over and over again, maybe thousands of times. In order to 
accomplish this, you are going to need the ability to call a procedure or 
function very fast.
 
That is what ILE/RPG is designed to do. Make maybe 10's of thousands or more of 
procedure calls and do it very fast. 
 
I recently had reason to run some timing tests to test the speed of a procedure 
call. 
 
I did 50 million procedures calls on a unloaded system in 27 seconds. That 
works out to about .00000054 seconds per call and that included passing by 
value. To say that the AS/400 and ILE/RPG can do procedure calls fast is an 
understatement. This contrasts with doing a subroutine call in .00000044 
seconds per call and a subroutine call is nothing but a jump to another 
address. IBM has worked intensively at optimizing procedure calls to make them 
as fast as possible and they are making them faster every release. 
 
In fact, if we look at the design of languages like ILE/RPG, they are designed 
to work this way. They are very efficient at running programs with many small 
procedures and local variables and very inefficient at running very large 
monolith programs with tons of static storage. 
 
My reason for bring this up is that, by necessity, implementing a solution that 
does not require using pointers requires using small functions and calling 
those functions many times. 
 
A concern has been stated that we must use pointers because we don't want to 
have the overhead of calling procedures over and over. 
 
As previously referred to, implementing a solution would probably include a 
function call to BuildAList followed by 1 to many calls to retrieve each piece 
of information. 
 
My point being here that this is exactly the kind of problem that ILE/RPG was 
designed to solve and do efficiently. In other words, worrying about a few 
hundred billionth of a second is a waste of time instead of just using the 
language the way it was designed. 
 
In contrast, the consumer of the service only needs to know what functions they 
need to call with simple examples and we can do error checking. If they try to 
call for record we have not built, we can issue an error and stop it. If you 
are receiving a data structure to load the record in, you can verify that it is 
the right size. If the internal structure changes, all caller that does not 
receive back all the new data is not going to fail if he does not need that new 
data. 
 
None of this is ideal. In an object oriented language we can return an object, 
something we cannot do in ILE/RPG but as a solution, it far outranges anything 
we have had to use before. 

Conclusion
 
Rather than using pointers, let's use the language the way the language was 
designed to be used. Returning pointers from service programs is almost never a 
good idea.