The CASA Release Process

Transcription

1 The CASA Release Process Nick Elias and the CASA Developers V July Introduction The two releases I have overseen since joining the CASA project have been eventful to say the least. Here is a brief summary of our existing release process: Nick Elias announces a feature freeze a few weeks before the release The developers diligently fix the regression tests while ad hoc testing is performed Errors, including new ones independent of the regression tests, are found and fixed The release occurs several (or many) weeks late The developers, now sufficiently stressed, are ready to face the next development cycle Repeat Clearly, we can do better. Our release process tax is very similar to the old build system tax. We spend a lot of time before each release trying to get it out the door. This time could be better spent (and possibly reduced) in a distributed manner, i.e., attempting to maintain viable stables (prereleases) throughout the development cycle. To create this document, I started with my own ideas, listened to the developers, modified my ideas as I saw fit, and organized them. The test system and the svn branches naturally mesh with the release process. In this version of this document, I also incorporate suggestions for the release process from the 2010 Developers Meeting. 1. The svn System The svn system has been expanded and regularized, consistent with industry (Table 1). Further, the test system has been more tightly integrated with the svn system. The combined systems now provide more flexibility for both test and release. For example, it is easier to create release candidates over regular intervals when there is an extra branch that allows for earlier testing. The developer builds are not part of the formal svn system, but they are mentioned for the sake of completeness. They act as private sandboxes for the developers. The active branch is the communal sandbox and formal entry point to the svn system. It is sometimes broken. Results are not guaranteed, so no JIRA tickets should be submitted based on runs from this branch.

2 The test branch has gone through light testing and is recommended for NRAO, ESO, and NAOJ employees. The stable branch corresponds to the current release candidate that can be used by experienced users everywhere. Systems personnel are the only developers allowed to check/move changes into the stable branch. During a feature freeze, however, only one designated systems person can act as gatekeeper for checking/moving changes into the stable branch. And, of course, the release branch is made available to the general astronomical community. The stable4dev branch, which is not part of the formal svn system, corresponds to the stable candidate where developers can check in last minute bug fixes during a feature freeze (in addition to checking them into active). The supervisor acts as the gatekeeper for stable4dev and makes the decisions on which changes can be checked in. Branch Executable Description Update Users name name timescale developer casapy Developer sandboxes ~ daily Developers active casapy (must run a script) Common sandbox Lightly tested branch Sometimes broken No guarantees on any results No JIRA tickets! ~ daily Developers; Internal illuminati who wish to review code changes very quickly test casapy test Moderately tested branch < weekly NRAO, ESO, and NAOJ staff stable4dev none Last minute bug fixes Separate from regular process Stable candidate feature freeze (two weeks before release) System leads stable casapy stable or casapy (via download) Heavily tested branch Release candidate ~ monthly NRAO, ESO, and NAOJ staff; Trusted visiting or external users release casapy Release (including patch) ~ biannually (release cycle) General astronomical community Table 1. The svn system branches and descriptions. The brown text indicates that those branches are not part of the formal system. Consider the desirable trends that arise naturally from this progression of svn branches. The amount of testing and the update timescale increases from the early to late branches. Conversely, the user expertise decreases from the early to late branches. 2. The Release Process The new release process integrates the svn branches as well as the unit and regression testing systems. It is designed to improve our final products and our overall efficiency. We spent too much time, effort, and sanity trying to get each release out. We also had difficulties with management of continuing development during feature freezes. The new release process is shown in Figure 1. The standard process (purple) takes place throughout each development cycle. As the name implies, the feature freeze process (blue) occurs during a feature freeze.

3 Figure 1. The new release process. The purple boxes represent the standard process. The blue boxes represent an additional feature freeze process immediately before a release. The purple and blue boxes are common to both processes.

4 New features and bug fixes must be checked into the active branch throughout a development cycle, including the feature freeze period. During a feature freeze, however, approved bug fixes may also be checked into the stable4dev branch because for possible inclusion in the next release. The supervisor is the gatekeeper for those changes. Systems personnel are the only developers allowed to move changes into the stable branch. During a feature freeze, however, only one designated systems person can act as gatekeeper for moving changes into the stable branch. This scheme has several distinct advantages. The stable branch allows us to implement more testing early in the development process, provide updates to external experienced users more quickly, and minimize the hassles of providing a release. During a feature freeze, development can continue, which means that all changes can move into the active and test branches (thus available to internal users for free human testing) without impacting upcoming stable and release branches. After the feature freeze has concluded, the test branch, which includes all new code changes, can immediately be vetted as the next stable branch. The separate stable4dev branch allows users to check in approved bug fixes for the stable and release without being affected by other development. The developer performs unit tests and a relevant regression test before checking new code into the active branch. The developer must specify whether the change requires extensive testing, which means that it must remain in the test branch for an extended period of time, perhaps beyond the next scheduled stable. The notation must be placed in the svn log and the JIRA ticket. For example, initializing an uninitialized pointer to prevent a segmentation fault does not require extensive testing. Modifying the CLI of a task, on the other hand, does. 3. The Development Cycle Schedule It is always a good idea to produce releases on time. To my knowledge, the CASA group has never successfully achieved this goal. The other extreme, producing a release only when certain features are implemented and tested, is quite radical and could lead to a highly irregular schedule. Therefore, I advocate quasi regular release dates unless they become untenable. At present, we attempt to put as many changes as possible into the code before a feature freeze. If important features are late, the release is delayed. This procedure has a major drawback, namely that other changes in the active and test branches languish for long periods of time. The new release process and development cycle schedule are designed to mitigate this problem. Delaying releases for important features serves no purpose, especially if we create stable releases on a regular basis. New features will not be implemented any faster if a release is delayed. The only reason to delay a release is to fix a serious bug (e.g.,

5 the serious spw issues that delayed the patch release). The stable4dev svn branch defined in the release process will be used to expedite last minute bug fixes. The supervisor and project scientist will determine if sufficient development has occurred for a release. Figure 2. Development cycle schedule and description. ~1m represents approximately 1 month. ~1w represents approximately 1 week. m.n.o represents the version number (major, release, stable). The release process was defined in Section 2, so it now must be scheduled. In Figure 2, I present a schedule for the development cycle. I assume that a new stable will be created every ~ every month., which will make releases less hectic Duties are described in the figure as well. Approximately five months plus one week are shown, allowing a few weeks of wiggle room to handle unexpected issues. 4.0 Testing What are your views on testing? Bryan Butler It s like voting. You should do it early and often. Nick Elias The above statements, which are only slightly paraphrased, occurred during one of my interviews for the CASA supervisor position. They neatly summarize my feelings on this matter.

6 We incorporated unit testing (more accurately, component testing) early in the development process, which should minimize the number of errors that reach the regression tests. Unit like tests from the regression testing system have been converted to unit tests. New unit tests using small datasets, which run very fast, have been created as well. More unit and e2e regression tests are on the way. In Table 2, I show the testing that must be performed for code to pass from one branch to the next. There is a lot of leeway, so tests can be moved around, changed, or added as needed. The unit tests have already caught simple errors before they reached the regression system, which has a longer completion time. I hope that this trend will continue and increase to the point where moving code from stable to release becomes almost effortless. Branch names Criteria to pass to the next level Who runs the tests? developer active Unit tests and one relevant regression test Developer active test Minimal low level regression tests (from Wes) Systems personnel Unit tests test stable Almost all e2e regression tests Systems personnel stable4dev stable Unit tests and selected e2e regression tests One systems person stable release Almost all or All e2e regression tests Systems personnel GUI tests Designated developer Table 2. The required tests to move along the svn branches. We already possess a significant number of unit tests. In addition to running unit tests (and a relevant regression test) before checking code into the active branch, developers are responsible for insuring that the unit tests cover the changes. If the unit tests are insufficient, the developers are also responsible for either for modifying an existing unit test or creating a new one. I do not expect a significant slow down in development. The number of new/modified unit tests will be small, and any time spent on new/modified unit tests will be recovered when bugs are found early. The regression test suite takes too long to run. On some machines, it takes more than a day. The upper limit should be less than one day, including required copying and generation of the HTML regression summary pages. We should review the regression tests to insure that there is no redundancy. If there is no redundancy, we should investigate the possibility of running the longer regression tests only before creating stable builds. It has been suggested that breakpoints (yielding intermediate results and/or intermediate output files) be inserted throughout the regression tests to help find problems. Code between breakpoints is approximately analogous to a unit test, so care must be taken to avoid duplication. This strategy has a low priority at this time, since we want to continue development and beef up unit testing. We must begin to incorporate EVLA and ALMA datasets as regressions. They will be large, and will place even more stress on our regression test system. I propose that

7 we reserve time on the NRAO cluster to run these tests on a weekly basis. Since most large datasets will be processed on clusters anyway, this strategy makes sense. When the CASA supervisor receives regression failure e mails, he often asks the assignees why the failure has occurred. This information is not formally recorded and tracked and it is sometimes difficult to determine if failures are serious and/or recurring. The test lead has created a spreadsheet that will help upper management and me to determine the status of failed regressions. The format resembles an excel spreadsheet (linked cells, tabs), but it is available via an HTML web page. There is a summary tab listing the latest status of each test. Each of the remaining tabs corresponds to a single regression test. The most recent line (top) of each tab is automatically linked to the summary page. Automated testing is a good thing, but we should never underestimate the exquisite ability of humans to break code, both CLI and GUI based. Like most academic development projects (and unlike most industrial development projects), we do not have a dedicated test team. We have ad hoc testers, but they are often busy and aren t available all of the time. Fortunately, the test branch will be available for internal testing. When an important feature is added, it would be nice if we could human test it quickly in a controlled manner. I have asked several scientists to help with testing, but as with the ad hoc testers they are busy as well. ESO has been running monthly tutorials to help their scientists to become more familiar with CASA. They have provided reports to the CASA supervisor after each session. I asked if they could test a list of specific bug fixes and new features, and they said yes to a limited number. I will continue to look for more volunteers to perform directed human testing. 5. The Closure Flowchart In the roadmap document, I divided up the entire development process into three flowcharts: assignment, development, and closure. I will not discuss the assignment and development flowcharts because they are not germane to this document and we already follow them pretty well anyway (tweaks to those flowcharts, which have a low priority, can be implemented in the future, if necessary). Taking the release process into account, I show a modified and simplified version of the closure flowchart (Figure 3). The color key is located in Figure 4. The entry point is in the upper left hand side, after the developer has finished development. The developer determines if the existing unit and regression tests are sufficient. If not, they must be augmented or added.

8 Assuming that the testing is successful, the developer puts the supervisor, project scientist, test lead, and systems lead on the watch list and resolves the ticket. The supervisor and project scientist are added to report work done to management. The Test lead is added so that she can keep track of regression tests. The Systems lead is added to inform him that a change is ready for testing and the eventual merging to the test branch. The developer then commits the changes to the active branch, including the JIRA ID and a note for extended testing (in the test branch, if necessary) in the svn comments. Figure 3. Modified closure flowchart of development process (originally from Roadmap document). The A connector returns to the development flowchart when a failure results. Figure 4. Color code for flowcharts. The flowchart of Figure 3 is meant for a single JIRA issue, but the systems lead does not typically perform his duties on single JIRA issue. When a new test branch is scheduled for creation (timescales less than a week), the systems lead performs unit tests and a subset of regression tests on a group of JIRA issues. All issues that pass are merged to the test branch, and the project scientist is added to the watch list of

9 all corresponding tickets. All issues that fail are returned to the developer. Upon successful merge to the Test branch, the supervisor closes the tickets. 6. Management Requirements Management has requested the following steps be performed before each formal release. 1. Management (Elias, Ott, Glendenning, and Butler) will have release readiness meetings where CASA status is discussed, including targets, documentation, and tests. 2. All unit and regression tests must pass before the release. Exceptions must be discussed/approved and listed in the release notes. 3. Early in the release cycle, ALMA and EVLA should send a list of features to be checked (ALMA: probably tutorial scripts; EVLA: probably archive). They have two weeks to supply the list. 4. The project scientist must approve task level documentation, release notes, and cookbook. Since the project scientist is added as a watcher on all resolved tickets, he will have the information he requires. 5. CASA management will announce to ALMA, EVLA, and scistaff that the stable before the expected release may be the one so that we can get additional testing. 7. Developers Checklist Use of the developers checklist is mandatory. When development is finished, including modification of tests, it must either be cut and pasted as a comment or attached as a separate file to the JIRA issue. The checklist: 1. Perform code development. [ ] 2. Are the unit and regression tests sufficient to test the changes? [Y/N] 3. If the unit and regression tests are insufficient, describe the modifications and/or additions. <Descriptions> 4. Have the unit tests and relevant regression test finished successfully? [Y/N] 5. If the tests have not finished successfully, return to code development (1).

10 6. Does the tool or task level documentation need to be changed? [Y/N] 7. If the tool or task level documentation needs to be changed, please do so. <Descriptions> 8. Add the supervisor, project scientist, test lead, and systems lead to the watch list. [ ] 9. Resolve the ticket. [ ] 10. Commit the code to the active branch. Please record JIRA ticket IDs in svn comments. Please add extensive testing statement to svn comments, if necessary. [ ] <Please record revision number(s) here> 11. If a feature freeze is in effect and if the changes are approved bug fixes (and no extensive testing required), commit the code to the stable4dev branch. Please record JIRA ticket IDs in svn comments. [ ] <Please record revision number(s) here, if different from #10> 8. Action Items Organize regressions such that the ones required to go from the test branch to the active branch take less than a day (Sandra, Nick, Ed,?). Schedule the stable branch creation dates (Nick, Wes, Dirk,?). Schedule the stable branch test sessions (Nick, Wes, Dirk,?). Schedule implementation of break points in regression tests (Nick, Sandra,?). LOW Priority. Schedule bug fix days based on old and recent JIRA tickets (Nick, Jürgen,?). Add EVLA and ALMA datasets to the regression tests (Nick, Sandra,?). Schedule EVLA and ALMA datasets for NRAO cluster (Sandra, Jonas, Nick, James, Kumar,?). Create list of new unit tests and prioritize (Sandra, Nick,?). Determine and implement scheme to reduce total time for regression tests to less than a day (Sandra, Ed, Nick,?).

11 Augment and assign viewer and plotms GUI tests (Nick, George, Darrell, Honglin, Daren,?). Find more testers; this is an ongoing task (Nick,?). Create a developers checklist document and put it on the CASA Index Twiki.