Yesterday I was running into a roadblock which unfortunately I still didn't clear today...

There’s a point in project development where you feel a bit like those war movie scenes. The hero is surrounded and is just fighting left and right; it seems hopeless, and it’s unclear how they’ll survive. This is where I am right now with DDTJ, deep in the swamp, with a hard to model problem.

Unfortunately, it doesn’t make for great writing. I can’t really communicate what I’m doing because a lot of it is just “shooting left and right”. The code I write is pretty bad since I’m trying to “get something working” and to be fair, I keep changing my mind. Hopefully, when I get something working, I’ll have the motivation to do a cleanup. But those are pretty hard to do for a spare time project.

One question I got is “why don’t you just use the JDI object model?”.

JDI (Java Debug Interface) already did all the work I’m doing. At first I felt it was over-designed but now I appreciate it more (it’s a tough problem). Unfortunately, I can’t reuse their object model. If I could, it would save me a ton of work. But their model isn’t the actual data. It’s tied to a remote VM state. So as the application continues its execution, the data is lost.

I need to replicate remote VM data locally so I can have it later when I want to generate a test case based on that data. That means doing something similar to what they did. But also pretty different. I need to transfer all the remote state information to my local VM.

That Doesn’t Sound too Hard

It isn’t when your object model only includes basic building blocks. But it becomes pretty hard when you need to mirror entire object graphs which are built from classes you don’t have in your local VM. If you’re thinking about fetching the classes to the local VM, that’s probably not the right idea.

Classes don’t work in isolation, you can’t just pull one class without pulling the entire application. Adding the entire application to the current VM is again... Not a good idea. It will break isolation and create a situation where we can affect execution significantly.

So the solution is to use the JDI API to take a full snapshot of the state with everything we’ll need later to generate the code. That also includes method/class meta-data. The complexity is mostly in the way we store the data and interact with this API. A lot of the code I write looks very “ugly”.

Dealing with Project Stalls

I made little progress today. I was debating posting this day at all, but I think it’s important that people read about the short-term failures as much as on the success.

Every decently complex project has a grunt/uncertainty phase. You still don’t know if you’ll make it on the other side of the tunnel and have something “good”. Unfortunately, there’s no real cure for that. But there are strategies to “manage” this and avoid a long setback.

Iterative Process

When you start a project from scratch, you’re effectively in front of an empty canvas. There’s nothing there and you can’t yet see your creation. Frustration and despair are common before you see something running. That’s why it’s important to aim much lower than your eventual goal.

We need to avoid “theorized” problems, such as performance optimizations, unnecessary features, etc. Stick to the MVP religiously, even at the cost of rewriting extensive parts of the code later on. You’re better off writing bad code and throwing it away if you reach an intermediate goal.

This maps very well to the concept of sprints. I’m not a big methodology fan and feel that people sometimes apply these processes too religiously. But the idea of aiming for short term quantifiable goals is powerful.

Shortcuts and Stopgaps

Hacks, or halfway points, are an excellent way to get something working. Right now, I’m mostly ignoring the test generation aspect of the project and only focusing on the data collection portion.

I also picked other shortcuts by avoiding a web interface and focusing only on the CLI for now. We can update later a lot of these decisions, but getting the first running version is crucial. Once you have things running, progress will speed up and compensate for any detours you took for these shortcuts/stopgaps.

It’s important to file issues for fixing such shortcuts/stopgaps. I don’t bother doing this on a single developer project (because I’m lazy). But if you’re doing this as part of your job, it's important. A good project manager would schedule some of your time for code cleanup and will make sure to revert to these issues.

Backtracking

This is the hardest thing to do. We sometimes take the wrong direction and spend too much time trying to get it to work.

It might be the “right direction” for the finished product. But for a sprint it might be too much. Unfortunately, once we started in a direction that seems “right” in the long term. It’s very hard to stop and throw away “correct code” in order to do what is effectively a hack.

This is something that sometimes works. It just takes a lot of effort. In this case, I’m conflicted if the data collection logic I’m trying to implement is over-engineered for this stage of the project. I can’t see a simpler solution that would actually work, but if there was a shortcut, I would take it...

One thing I might need to backtrack is my focus on quality and testing. I think I might need to put the quality aspect on hold and ignore code coverage for now.

Next Week

I was hoping to have something running this weekend as a Christmas present... These hopes crashed against the harsh reality and constrained time because of family obligations. I have a feeling I would have made it if I was working at an office.

I hope to get data collection working early next week, otherwise code generation just wouldn’t happen and the whole experiment would tank. I’m not sure when I’ll be able to get time off work to finish such an extensive project if I don’t do this now, so this is pretty crucial.

In retrospect, for this week I feel I spent too much time on coverage and quality, which was probably the wrong thing to do at this early stage. Usually, I only add these tools later on when the project is functional.

I’m holding off on this decision. I want to see how next week starts and where I’m at.

If you find this series interesting/useful feel free to follow me on twitter...