Thoughts on Software Engineering Project Management

I've been working as a product manager/project manager in startup software companies for the past 5 years. In this type of role you always get squeezed between sales/marketing and engineering. There are lots of ideas about how people should work together floating around out there, but getting objectively good results is more art than science. Here are some things I've learned recently, in no particular order.

The fundamentals

Build a prototype, don't mix prototype with MVP

For new prodouct introduction (NPI) in newly formed companies, most people--let's call them product owners, because there are always several--don't know what they want. Even if they did know what they wanted, they wouldn't know how to communicate it. It might not even be their fault; they might have really done a lot of planning, talking to potential users/customers, and creating written documentation. But they get new information all the time, and they are naturally persuasive (otherwise they wouldn't be doing the jobs they are doing). So you have some money and a team of engineers, but no product, no customers, a short delivery timeline, and shifting requirements.

The problem becomes more acute if these people have heard of modern agile/scrum frameworks. They think that they are welcome to change the software every week, or even every day. That's what "agile" and daily standups are for, aren't they? To give the product owner the chance to change the scope of the project whenever they feel like it! What ends up happening with these frameworks is you end up with a product which is continuously evolving and everybody is continuously unhappy. For many years software teams have been promoting agile and scrum. From what I've seen these only help deliver results in cases where programmers are working together, not for programmers working with project managers or anyone else. If the framework is introduced as a project management tool, then this is just a clever way to get perpetual funding. Maybe that works for consultants selling in big corporate environments, but certainly this is not ideal for startups or companies that need to deliver results in accordance with a budget.

I think the problem can be more clearly framed in terms of product development. A good solution is to cleanly separate the prototype from the minimum viable product (MVP) at the very beginning of the venture or NPI planning process. These are two separate projects. This distinction is important for two reasons:

  1. Fail fast, fail cheap. If you want to throw something at a potential customer and say "if we made this and it did that, and priced it like this, would you buy it?", then you want a prototype. A prototype ideally should only take a few days or weeks to deliver. A prototype is the best way to communicate between engineering and sales teams. Not written reports, not chat groups, not meetings!
  2. Minimize or eliminate technical debt if the prototype passes the crux. If you get a bunch of people saying yes to a prototype and now you need to create the actual product, you're going to need to make very different assumptions about the work that needs to be done. You might choose a different framework, architecture, or programming language. If you always try to transform or extend the codebase you used for the prototype, you will maybe deliver your MVP, but you will have a lot of problems with refactoring, and they will only get worse over time. Think it through carefully first, in most cases you need to toss the prototype into the bin as soon as it's time to move towards an MVP.

I think this distinction is important for any NPI project, regardless of who you are asking to fund it, be it a VC or a VP. It's tricky to demonstrate the potential value of a thing, because if it doesn't exist yet, how can you prove that people will actually use it? And nobody wants to invest big money or effort just to try to demonstrate that people will use it. You always want to go through that process as quickly and cheaply as possible. The best way to do that is to separate Prototype from MVP.

Event data capture, logging, and unit tests from the beginning

You don't need to pay $1'000 a month for Heap. But you do need to collect event data wherever you can. Treat this exercise as another form of logging. Even if you just use a modified logging system and then dig through the the logs as a poor-man's Heap, that's better than nothing. If you don't collect event data, then you won't have a chance to learn much, and the project will die. Beware survivorship bias when analyzing this data.

Everyone builds webapps using Django, Node, etc. and these frameworks all already have facilities for logging. Make sure to use them, so debugging is easier. And make sure to write unit tests as you go, don't put this off until later. If you treat adding unit tests as a separate project then you'll probably add 50% to the amount of time your project will take to get to the same level of quality if you had just done unit tests from the beginning.

Use a Gantt Chart

Classic project management is a winning formula. Ignore the agile/scrum consultants who poo-poo "waterfalls". Waterfalls are great. People travel great distances to stand in awe of a waterfall, to hear its roar, and to enjoy the cool mist it gives off, and to swim around in the pool that forms at the bottom on a warm summer day. For harmony and progress in a project, you need to set milestones. Get consensus agreement on those milestones, and then work towards them. But changing milestones is a pain! Well, yes, of course. You don't want to change milestones! The key is to try to break large projects into small projects lasting just a month or two, so the effort required to change milestones is relatively large compared to simply letting the project complete as expected or to cancel the project altogether.

Another complaint I have about agile/scrum in practice is the way these often treat programmer time as an unlimited resource. Regardless of whether you have salaried people on a project or not, you need to allocate people's time as a resource. It's ok to have some people on a project to be underutilized sometimes! Having some time to think and mentally reset is important to making progress. The important thing is to be able to have an idea if the project is on track or not, not whether (necessarily) any one person has spent more or less time on the project than originally planned.

This is where a Gantt chart comes into play. With one look, you can show your complete project and its status, and it immediately relates back to what you pay vs what you get. You shouldn't use a Gantt chart to investigate who is spending time on what, because that doesn't work with software development. That's the wrong way to use this tool, and I imagine that people using it in this way is what led to this agile/scrum hype. Anybody can create a Gantt chart easily using free and open source software like GanttProject.

Use Business Process Modeling Notation (BPMN)

BPMN is another important cross-organization communication tool. This lets you model business processes, like payroll, or customer support. The first time you see BPMN you might think it looks like a normal diagramming tool, like But what it really does is help you go from modeling a process to turning that model into computer code (e.g. java) without much fuss. The strict conventions will force you to really think through your process, and you won't be able to avoid seeing all the inefficiencies! You'll also be tempted to try to fix these inefficiencies right away...

I'm using Bonita BPM, which is free and open-source, but there are many different implementations out there. You can also use less strict diagramming tools, but I think it's worth making the extra effort in order to be able to communicate more efficiently between business and engineering. If you're in a startup environment, you first need to ask "what do we need to get done?" immediately followed by "how do we get it done?". As soon as you are able to answer the first question as a team, you should move to BPMN models to answer the second.

Setup and use Databases

Bigger companies already have this figured out, and they spend big money to have consultants from SAP, Oracle, and so on setup custom databases for them. If you already have something setup, great! But many startups and small companies try to survive using things like dropbox, google drive, and so on. This is a big mistake! Taking the time to plan ahead, to develop your own databases and tables and methods for interacting with them, will save you a lot of trouble. It could even be the difference between your company's success and failure. If you believe that people are disciplined enough to use shared filesystems without a structured way to input/output data, I've got a bridge to sell you.

Even the smartest, most disciplined and diligent people will still put things in the wrong place, in the wrong format, overwrite the wrong file, or just forget to persist a file in the first place. You need to have some sort of structured way to input data, update that data, and retrieve that data. It doesn't have to be fancy, a simple PostgreSQL database with a command line interface. If you have time, create a GUI. Let people SSH into the server where your application is running. Also take the time to setup a method to backup your database - at this point you could push an encrypted backup to a cloud solution like Google Drive or Dropbox.

After you have your process clearly described using BPMN, you should be able to easily create custom databases/tables and interfaces. Depending on how clever you are (or how much pain you can endure) you can even convert your BPMN process models directly to an application for storing/processing business data. But I imagine most people will prefer to keep these as separate projects, and not try to link them. After all, not everyone is wanting to write all of their business software in Java.