Python: Fix header field values character encoding.

On GitHub, @RomainMou reported an issue whereby HTTP header field values
where being incorrectly reported as non-ascii by the Python .isacii()
met

Python: Fix header field values character encoding.

On GitHub, @RomainMou reported an issue whereby HTTP header field values
where being incorrectly reported as non-ascii by the Python .isacii()
method.

For example, using the following test application

def application(environ, start_response):
t = environ['HTTP_ASCIITEST']

t = "'" + t + "'" + " (" + str(len(t)) + ")"

if t.isascii():
t = t + " [ascii]"
else:
t = t + " [non-ascii]"

resp = t + "\n\n"

start_response("200 OK", [("Content-Type", "text/plain")])
return (bytes(resp, 'latin1'))

You would see the following

$ curl -H "ASCIITEST: $" http://localhost:8080/
'$' (1) [non-ascii]

'$' has an ASCII code of 0x24 (36).

The initial idea was to adjust the second parameter to the
PyUnicode_New() call from 255 to 127. This unfortunately had the
opposite effect.

$ curl -H "ASCIITEST: $" http://localhost:8080/
'$' (1) [ascii]

Good. However...

$ curl -H "ASCIITEST: £" http://localhost:8080/
'£' (2) [ascii]

Not good. Let's take a closer look at this.

'£' is not in basic ASCII, but is in extended ASCII with a value of 0xA3
(163). Its UTF-8 encoding is 0xC2 0xA3, hence the length of 2 bytes
above.

$ strace -s 256 -e sendto,recvfrom curl -H "ASCIITEST: £" http://localhost:8080/
sendto(5, "GET / HTTP/1.1\r\nHost: localhost:8080\r\nUser-Agent: curl/8.0.1\r\nAccept: */*\r\nASCIITEST: \302\243\r\n\r\n", 92, MSG_NOSIGNAL, NULL, 0) = 92
recvfrom(5, "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nServer: Unit/1.30.0\r\nDate: Mon, 22 May 2023 12:44:11 GMT\r\nTransfer-Encoding: chunked\r\n\r\n12\r\n'\302\243' (2) [ascii]\n\n\r\n0\r\n\r\n", 102400, 0, NULL, NULL) = 160
'£' (2) [ascii]

So we can see curl sent it UTF-8 encoded '\302\243\' which is C octal
escaped UTF-8 for 0xC2 0xA3, and we got the same back. But it should not
be marked as ASCII.

When doing PyUnicode_New(size, 127) it sets the buffer as ASCII. So we
need to use another function and that function would appear to be

PyUnicode_DecodeCharmap()

Which creates an Unicode object with the correct ascii/non-ascii
properties based on the character encoding.

With this function we now get

$ curl -H "ASCIITEST: $" http://localhost:8080/
'$' (1) [ascii]

$ curl -H "ASCIITEST: £" http://localhost:8080/
'£' (2) [non-ascii]

and for good measure

$ curl -H "ASCIITEST: $ £" http://localhost:8080/
'$ £' (4) [non-ascii]

$ curl -H "ASCIITEST: $" -H "ASCIITEST: £" http://localhost:8080/
'$, £' (5) [non-ascii]

PyUnicode_DecodeCharmap() does require having the full string upfront so
we need to build up the potentially comma separated header field values
string before invoking this function.

I did not want to touch the Python 2.7 code (which may or may not even
be affected by this) so kept these changes completely isolated from
that, hence a slight duplication with the for () loop.

Python 2.7 was sunset on January 1st 2020[0], so this code will
hopefully just disappear soon anyway.

I also purposefully didn't touch other code that may well have similar
issues (such as the HTTP header field names) if we ever get issue
reports about them, we'll deal with them then.

[0]: <https://www.python.org/doc/sunset-python-2/>

Link: <https://docs.python.org/3/c-api/unicode.html>
Closes: <https://github.com/nginx/unit/issues/868>
Reported-by: RomainMou <https://github.com/RomainMou>
Tested-by: RomainMou <https://github.com/RomainMou>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>

show more ...

Python: Do nxt_unit_sptr_get() earlier in nxt_python_field_value().

This is a preparatory patch for fixing an issue with the encoding of
http header field values.

This patch simply moves the nxt_un

Python: Do nxt_unit_sptr_get() earlier in nxt_python_field_value().

This is a preparatory patch for fixing an issue with the encoding of
http header field values.

This patch simply moves the nxt_unit_sptr_get() to the top of the
function where we will need it in the next commit.

Signed-off-by: Andrew Clayton <a.clayton@nginx.com>

show more ...

Python: Added "prefix" to configuration.

This patch gives users the option to set a `"prefix"` attribute
for Python applications, either at the top level or for specific
`"target"`s. If the attribut

Python: Added "prefix" to configuration.

This patch gives users the option to set a `"prefix"` attribute
for Python applications, either at the top level or for specific
`"target"`s. If the attribute is present, the value of `"prefix"`
must be a string beginning with `"/"`. If the value of the `"prefix"`
attribute is longer than 1 character and ends in `"/"`, the
trailing `"/"` is stripped.

The purpose of the `"prefix"` attribute is to set the `SCRIPT_NAME`
context value for WSGI applications and the `root_path` context
value for ASGI applications, allowing applications to properly route
requests regardless of the path that the server uses to expose the
application.

The context value is only set if the request's URL path begins with
the value of the `"prefix"` attribute. In all other cases, the
`SCRIPT_NAME` or `root_path` values are not set. In addition, for
WSGI applications, the value of `"prefix"` will be stripped from
the beginning of the request's URL path before it is sent to the
application.

Reviewed-by: Andrei Zeliankou <zelenkov@nginx.com>
Reviewed-by: Artem Konev <artem.konev@nginx.com>
Signed-off-by: Alejandro Colomar <alx@nginx.com>

show more ...

Refactored functions that set WSGI variables.

Splitting `nxt_python_add_sptr` into several functions will make future
additions easier.

Signed-off-by: Alejandro Colomar <alx@nginx.com>

Renamed a couple of members of nxt_unit_request_t.

This is a preparatory patch that renames the 'local' and 'local_length'
members of the nxt_unit_request_t structure to 'local_addr' and
'local_addr

Renamed a couple of members of nxt_unit_request_t.

This is a preparatory patch that renames the 'local' and 'local_length'
members of the nxt_unit_request_t structure to 'local_addr' and
'local_addr_length' in preparation for the adding of 'local_port' and
'local_port_length' members.

Suggested-by: Zhidao HONG <z.hong@f5.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>

show more ...

Python: using default event_loop for main thread for ASGI.

Unit's ASGI implementation creates a new event loop to run an application for
each thread since 542b5b8c0647. This may cause unexpected ex

Python: using default event_loop for main thread for ASGI.

Unit's ASGI implementation creates a new event loop to run an application for
each thread since 542b5b8c0647. This may cause unexpected exceptions or
strange bugs if asyncio synchronisation primitives are initialised before the
application starts (e.g. globally).

Although the approach with a new event loop for the main thread is consistent
and helps to prepare the application to run in multiple threads, it can be a
source of pain for people who just want to run single-threaded ASGI
applications in Unit.

This is related to #560 issue on GitHub.

show more ...

Python: support for multiple targets.

Python: WSGI environment copying moved out of request processing.

The WSGI environment dictionary contains a number of static items, that are
pre-initialized on application start. Then it's copied

Python: WSGI environment copying moved out of request processing.

The WSGI environment dictionary contains a number of static items, that are
pre-initialized on application start. Then it's copied for each request to be
filled with request-related data.

Now this dictionary copy operation will be done between processing of requests,
which should save some CPU cycles during request processing and thus reduce
response latency for non-peak load periods.

show more ...

Python: request processing in multiple threads.

This closes #459 issue on GitHub.

Fixed building with Python 3.9.

PyUnicode_GET_SIZE() in deprecated since 3.3 and will be removed in 3.12.
In version 3.9 it was explicitly marked by deprecation warning causing
compilation error wit

Fixed building with Python 3.9.

PyUnicode_GET_SIZE() in deprecated since 3.3 and will be removed in 3.12.
In version 3.9 it was explicitly marked by deprecation warning causing
compilation error with Unit.

PyUnicode_GET_LENGTH() must be used instead.

show more ...

Python: ASGI server introduced.

This closes #461 issue on GitHub.

Python: changed request headers format in router protocol.

The coming ASGI support requires raw HTTP headers format. Headers grouping
and upcase code were moved to WSGI module.

Python: split module initialization from WSGI implementation.

This is required for futher ASGI implementation.

Python: source file moved to 'python' sub-directory.

No functional changes. Get ready for an increase in file number.