Share on Facebook
Share on Google+
Share on Twitter
Share on LinkedIn
An eventful coding week for Simple TCP proxy
I've been mailing a bit on the python-users mailing list about STP this week, and there were some discussions there that were unfruitful - but I got some useful tips in the process:https://mail.python.org/pipermail/python-list/2022-July/9070...
This TCP proxy needs to scale and perform reasonably well, so the last thing I worked on was making sure that downloading large files via the proxy works reasonably well, in these commits:
https://github.com/morphex/stp/commit/3328a45d5314040abd37fa...
https://github.com/morphex/stp/commit/5e477bf5756e2763db799c...
In was interesting, and relieving to see that I could speed up the download process 3-4x by multiplying the "transfer buffer" by 8. A local download of the test.tar file was around 1.3 GB/s via lighttpd, and 122 MB/s via STP. After the buffer change, the speed increased to 449 MB/s. Which is more than what a gigabit connection to the internet can handle.
So, so far so good, and good enough.
The previous commit:
https://github.com/morphex/stp/commit/5e477bf5756e2763db799c...
I got a tip for, and reduced the thread stack size to the minimum, which is from what I gather, necessary to ensure that it will run reliably on different platforms - as Linux has a rather big thread size and that could break on others if you're reliant on it.
I also implemented a "fall through", so that the thread sending data back and forth doesn't stop and wait for .N number of seconds, if data has just been sent back and forth.
And in the commit before that, I adjusted the sleep time for threads "polling" on whether or not they have become active:
https://github.com/morphex/stp/commit/9910ca8c80e9d150222b68...
which also increased the speed at which STP could deal with the connection queue.
There has been some questions on the Python list as to exactly what I'm building and why, but I think a fair description would be that it is a simple to understand, and run, TCP proxy which does some useful things out of the box.
Coming from the web development and hosting world, I'm tempted to add some HTTP protocol features as well, but that's something that belongs in another project, maybe using STP as the frame which it plugs into.
[Permalink] [By morphex] [A simple TCP Proxy (Atom feed)] [31 Jul 20:08 Europe/Oslo]
A simple TCP Proxy
So as I was working on the SMPS, simple message passing system, I looked at ways of protecting it on the public net.Port knocking was one thing I looked at, and although it looked fine, I thought this was an opportunity to hone my Python/programming skills, and freshen up a bit on programming.
So I ended up writing a simple TCP proxy, and a working copy is here:
https://github.com/morphex/stp/tree/e781e84eee7b510879593d1a...
The test_http.sh script will setup a proxy in front of a local webserver.
The STP seems a bit sluggish when a lot of connections are made, but I'll have a look at how to optimize that performance.
[Permalink] [By morphex] [An SSL TCP client/server implementation in Python (Atom feed)] [26 Jul 23:52 Europe/Oslo]
Encountering Python crash on numerous client connects to socketserver
On the SMPS package, I've refactored a bit since the last blog post, and I also encountered what looks like a bug in Python.Python can randomly crash if too many client connections are made to the socketserver, and it crashes in such a way that it is difficult for me as a regular Python programmer to understand what the problem could be.
So although I was tempted to do a bit of C debugging and poke around, I do have limited time and energy, so I did the smart thing and patched it up so that it is easy to reproduce the problem:
https://github.com/morphex/smps/commit/59fbc78ebddcc3a68c6ea...
and then I created an issue in the Python bug tracker:
https://github.com/python/cpython/issues/95204
[Added later:] Aha. So it turns out I was quitting threads, and this gave an unclear error message. Thanks to Jeff Epler for pointing that out.
That got worked on in this commit:
https://github.com/morphex/smps/commit/9b539075627dd2016fca6a98f70913f21671e072
and the parent commit. Now with > 400 test connections I get an SSL handshake timeout, but, that's in ssl.c and probably due to some configuration setting somewhere.
[Permalink] [By morphex] [An SSL TCP client/server implementation in Python (Atom feed)] [24 Jul 18:13 Europe/Oslo]
Thread-safe message database for SMPS
I hacked a little more on the SMPS project today, which resulted in the following commit:https://github.com/morphex/smps/commit/8c565c561c0a9593f4d5e...
A pretty straight-forward message passing queue, that expires messages after n number of seconds. I guess the only thing left now before it can be put into practical use, is adding some sort of handshake with a secret, so that it is possible to protect the message queue from unauthorized access.
[Permalink] [By morphex] [An SSL TCP client/server implementation in Python (Atom feed)] [18 Jul 17:37 Europe/Oslo]
An SSL TCP client/server implementation in Python
Related to work I found myself needing a solution to pass messages, so I started working on something simple, just to get the basics down.I started out with the idea of doing it over HTTPS, but after a bit, I decided to go for a more minimalistic solution, using just TCP over SSL.
I've implemented a simple example of how to implement an SSL protected client/server message passing solution in Python 3.9. I googled a bit, and couldn't find a good, solid, simple example, so I figured the first iteration of the Simple Message Passing System could serve as an example, for others as well.
The committed state of the files [edit: for that first iteration are] here:
https://github.com/morphex/smps/tree/5e519cbbdf254c09ae3fe4d...
[Permalink] [By morphex] [Python and web (Atom feed)] [17 Jul 20:33 Europe/Oslo]