A workflow for handling multiple concurrent changes on a single software system

To me, a change will mean that I am going to have more rice on my bowl. And in my nature of work, changes to the software systems under my care occur pretty often. In fact, changes had benefited me, as changes serve to increase my company's competitiveness so that she can continue to buy me my daily bread.

There was this time when I was working with a software development house which my company had employed to build our new system. Nearing project completion, I volunteered to engineer a new workflow for my company to handle software development after we took over the system from the software development house. The main objective of the workflow is for coordinating change deliveries from multiple development teams on a single software system suite.

This post documents the workflow that I had engineered. When there are notable improvements made to this workflow, I shall return to this post to include them.

Phases in our new Software Development Life Cycle (SDLC)

In my new workflow, I define four phases after requirements had been gathered from stakeholders of the system:

1. The Development phase
2. The User Acceptance Testing (UAT) phase
3. The Staging phase
4. The Production phase

The following table portrays how they relate to one another.

Development	User Acceptance Testing
	Staging
Production

The objective for any change request is to appear in the Production server(s). Code changes needed for the change request happens in the Development phase. The Development phase spans over the User Acceptance Testing (UAT) and Staging phases. After changes had been finalized in the Staging phase, there must be no further code changes for the system to proceed to the Production phase.

Walking the development phase

The development phase is a mandatory phase in order for changes to be realized on the software system. Every developer in the team must utilize a single source control server to synchronize their development work.

The source control server will manage three categories of code bases (which I had adapted from SVN parlance):

1. A single trunk that contains codes for creating the build artifacts for the staging and production environment.
2. Multiple branches that contains codes for creating the build artifacts for conducting user acceptance tests on the various change requests.
3. Multiple tags that contains codes for creating the build artifacts of stable releases of the software system for rollback purposes.

Performing development on branches

When a new change request is received, a branch shall be created from the trunk. Developers assigned to work on that change will perform changes to the code base at that new branch. Any changes made at the branch will not take effect on the trunk, hence production pushes will not be affected by changes in the branches.

Performing development work on the trunk

Development work on the trunk can only proceed if there are branches that are ready to be upgraded to production state. When branches are ready, the developers who worked on them will merge them to the trunk. When merging work had begun on the trunk, new merger of branches cannot be allowed - until the trunk is marked as healthy.

Nobody shall be allowed to commit complex code changes directly onto the trunk.

Using tags as snapshots of healthy software system state

Before development work starts on the trunk, we shall create a tag to keep track of the healthy state of the trunk. There is no development work to be performed on tags; tags are for the purpose of restoring the production software system to a previous stable version. Tags are therefore necessary to ensure that we have the agility to hide critical bugs from the production system while our developers are given ample time to fix the bugs on new branches.

Confirming change fulfillment with users through the User Acceptance Test (UAT) phase

When the developers are confident with the development work that they had done to fulfill a change request, they will create a build artifact from the respective branch and deploy the build artifact to a dedicated server instance. The users who had requested for that change will then be invited to that server instance to check whether the change is correctly implemented by the developers.

When the users are happy with their changes, the branch will be ready for merging to the trunk.

Verifying the health of the trunk with the Staging phase

The Staging phase will happen when there are one or more changes are ready to appear on the Production server(s). Developers who worked on the branches will then be allowed to merge their changes onto the trunk. When merging is done by all concerned developers, a build artifact will then be created from the trunk and deployed onto a staging server. The staging server should resemble the production server(s) as much as possible in order to weed out bugs that are caused by behavioral between different server makes from the UAT and production environment.

Some may feel that more agility can be achieved by bypassing the Staging phase and merging branches directly onto the Production system. However, bugs can surface during code merge and they risk introducing those bugs in the production system.

Pushing the healthy trunk to production state in the Production phase

After some quality assurance had been done on the staging server, the trunk will then be deemed as healthy and ready for production deployment. The build engineer will then create the build artifact from the trunk and deploy it to the production server(s).

Operating procedures for change request fulfillment

The categorization of different code bases and SDLC phases set a basis for operating procedures to be formulated. Whenever a change request appears in our backlog, we shall classify it as either cosmetic or complex.

Cosmetic change

A cosmetic change could be a change in marketing collaterals or a change in that text label that ask for the user's name. To bring such a change to production, we should follow the following operating procedure to bring the change to production:

1. Create a tag from trunk
2. Develop changes directly onto trunk
3. Build the trunk
4. Deploy build artifact on staging server
5. Test and verify changes on staging server
6. Deploy trunk on production server(s)

Complex changes

A complex change usually involves database changes. It could be an entire module for enabling our users to chat with one another or a new algorithm that predicts the buying trend of our users. When we receive such a change, we should follow the following procedure to bring the change to production:

1. Make a branch from the trunk
2. Reserve a UAT server instance for deployment code changes for user acceptance
3. Develop changes on the branch
4. Build the branch
5. Deploy build artifact on assigned UAT server instance
6. Conduct user acceptance tests on UAT server
7. Make a tag of the trunk
8. Merge branch to trunk
9. Build the trunk
10. Deploy trunk artifact onto staging server
11. Test and verify on staging server and conduct second round of user acceptance tests (if possible)
12. Deploy trunk onto production server(s)

• Share
• Tweet
• Share
• Save