The last release of YAWAST was on January 1, 2020; while the release history was sometimes unpredictable, the goal was a new release each month with new features and bug fixes. I intentionally took January off from the project. In February, I left the company I was at; the team of penetration testers there had helped to inspire new features while looking for ways to make them more productive. But something else happened in February, an issue was opened – something that appeared to be simple, but in fact, made me realize that the entire project was in doubt.

While I first mentioned YAWAST here in the fall of 2015, I had actually started working on it back in 2013. At the time, I was looking for something interesting to work on to learn Ruby, and YAWAST, or “Yet Another Web Application Security Tool” was born. I worked on it from time to time, adding new features that would address things I wanted. Eventually, it evolved into something more, something better, and had become something useful.

In the spring of 2019, I became somewhat frustrated at the lack of community involvement; there were a surprising number of users, but minimal code was being written by anyone but me. One thought I had was that it was the language that was holding people back from trying to fix issues or add new features on their own, so in May, I created a poll to learn more about the impact of language:

Poll: Would you be more/less likely to contribute to an open-source security tool based on the language it's written in? If so, which language would lead you to be most likely to contribute? (Please RT, trying to get a feel for what security devs prefer - Thanks!)

— Adam Caudill (@adamcaudill) May 18, 2019

This gave a clear answer: as suspected, the community had moved on from Ruby to Python as the preferred language. The poll also confirmed that most people are more likely to contribute to a project based on the language that it’s written in. While this was a small poll, very informal, and likely had a huge margin of error, it was also the information I had to work with (I’ve written more about the rewrite here & here).

The last Ruby version of YAWAST was released on May 13, 2019, and then the rewrite began. Over the next 94 days, YAWAST was rebuilt from scratch in Python. Nearly every day, I worked on it for at least a couple of hours, though most days it received every minute of free time I had – on weekends, I would spend 16+ hours a day. It wasn’t a simple project or a small time investment. Everything was changed, a new architecture, a new way of building features, better performance (in certain places, worse in others), more flexible options, and a ton of new features. Not only was it rewritten, but it was also greatly improved.

It was fairly easy to deploy and use on Windows, macOS, and Linux – on Kali Linux, it worked with almost no setup. It was beautiful.

Even before I posted that poll, I was almost certain that I would rewrite it, moving away from Ruby to something that offered me better options. I had started running into too many limitations and frustrations due to the architecture & design decisions that came with Ruby. Ruby was wasting too much of the time I could dedicate to it. The main question was, what language do I go with? There were a number of options I considered:

• Python – This had become almost the de facto language of the security community. It was easy for developers of any experience level. There was good cross-platform support and tooling available, reasonable debugging support, and a nice library of open source components that would help to speed up development.
• C# – I had spent years working in C#, and the cross-platform support has come so far in recent years. There’s great tooling on Windows and reasonable tooling available for other platforms, a vast standard library, it’s a breeze to debug, and still fairly easy for developers to work with. What it was lacking was any support in the security community.
• Go – A great standard library, reasonable tooling, a not great deployment story, but great cross-platform support. The deployment issues hurt it, as did one of it’s better features: the great standard library often errs of the side of making it hard to do things in an insecure manner, which is something of a problem when developing a security testing tool. This focus on dropping support for insecure options would require extra work, and there was only so much time that could be invested in it.
• Rust – Fast, easy deployment. Tooling options aren’t great. The standard library is so tiny that almost everything has to be done with third-party libraries or in custom code. It’s also one of the hardest options for developers that are new to it.

Given that one goal was to gain more activity from the community so that I wasn’t a single point of failure for the project, languages that are easiest to work with have an obvious advantage. All things considered, Python won.

There were some issues that I didn’t weigh though, for example, the threading / parallel processing support became a huge pain during development and resulted in some unpleasant design decisions to make it all work. Not since Visual Basic 6 have I worked with a language that made simple threading so painful or annoying. The Global Interpreter Lock (GIL) presented an entirely unreasonable challenge to achieving reasonable performance and cost me a ridiculous number of hours while trying to work around the limitations that it imposed. I came into the project knowing that it would be an issue, but I didn’t realize how intractable the issue would be.

A seemingly simple issue turned into an existential threat as I researched the issue and looked at solutions. I was soon plunged into a dependency hell, not of my own making, but made by various others. It started with Kali Linux changing the version of Python they were shipping, which broke a third-party dependency, which led to looking into a variety of options. Some options were easy but created other issues; others worked but made deployment too complicated. The easy way of installing on Kali would result in breaking a number of other applications – certainly not an option. The complex options were just that, complex – complex enough that I didn’t want to ask users to do it.

While researching this, more compatibility issues surfaced, and fixing the easiest install methods became impossible. The complex set of interdependencies had become too much for Python’s package management system, especially on Kali with its wide variety of other tools installed. The Python version of YAWAST was released during a window when all of the stars were aligned; everything just worked as hoped. By February, just a few months after being released, the environment had changed enough that maintaining compatibility became a major problem.

This seemingly simple issue turned into weeks of research, testing, building and rebuilding virtual machines, testing forks of dependencies, building install scripts, and trying to identify other viable options. The goal was a future-proof solution that was easy for users and reasonable to maintain. While I’ll not bore anyone with the technical details, I will sum up my findings: the only viable deployment option for a complex Python application is a container.

For YAWAST, the Docker image provided was the only option that would be fairly easy for users and be fairly future proof. Installing it directly on a machine that wasn’t set up just for it was always going to be an issue and could break at any time – there’s no way to future-proof it, no way to ensure that it’ll be reliable in the future.

Python was intended to provide a better path forward for YAWAST; instead, it may well be the doom of the project.

Several factors are weighing on the future of YAWAST:

• Requiring the use of Docker for a fairly small application seems to be an unreasonable burden for users; in addition, Docker is increasingly falling out of favor, with other options becoming more popular.
• Some parts of YAWAST are far slower than they should be due to a combination of the GIL and the application’s architecture. This isn’t something that can be fixed without a substantial time investment.
• There are no active developers on the project; I’ve not been active this year, and there are no others active today.
• I’ve debated embarking on another rewrite of YAWAST, likely in Go or Rust, though the time required would be hard to justify.
• My work has shifted from offensive to defensive, and from being responsible for testing a large number of web applications each year to defending a handful of well-built and well-defended applications.

The future of YAWAST isn’t decided, the current version may be the last ever to be released, or it may gain a new lease on life. As a project, the future is in question.