YAJL implementation / suggestions

Hi Scott,



I have debugged the error gio describes in another tread and there seems to
be a storage allocation problem in YAJL.



When I reach approx. 9-10 MB in the JSON build YAJL fails and doesn’t
extend its buffer. It would be nice if you were able to specify a standard
heap size of the output buffer in yajl_genOpen(*ON:100000); that then was
used as the initial heap size and afterwards was used to extend the
allocated storage for the generation instead of having to modifying your
YAJL distribution that would affect all YAJL generations and thereby
overloading allocated storage even if the program only produces small JSON
objects.



Another thing is the yajl_copyBuf(); that returns data to a local
field/allocated storage and a length of the converted result from UTF-8 to
EBCDIC. This is not only an unnecessary conversion (I know you can pass
1208 as the CCSID) but it also requires that you allocate a field/storage
beforehand. In other words this has to be big enough to hold the unknown
length of the result.



My suggestion is a sub-procedure like yajl_getBuf(addr:size); that calls
yajl_gen_get_buf(); and just returns the UTF-8 address and size to the main
program. This information can then be used to pass the generated UTF-8
result with either text/… charset=utf-8 or any other non “text/…” mime
header so Apache doesn’t convert back from EBCDIC to UTF-8 and thereby
speeding thing up directly or used to allocate local storage that fits the
JSON result.



The first I don’t know how to do, the second I could do myself but that
would require that I modify your YAJL implementation in my local copy or
you implement the simple sub-procedure (yajl_getBuf) in yours.