Organization

Location

Hiomotie 5, Helsinki

Project

Upgrade the Intra Task Force set and allocated by Kalle Tarpila

Background

what to do and how to do: N4DSCM.ppt/1/Dec/1998/Kta

Schedule

By 1999-01-30

Participants

NMS For Data R&D members

Version information

Version / status of this review report: 1.0 / Ready.

[Note by CBo: Original Word document issued 01/02/1999]

Subject

• Software Process and Tools consultancy effort by Rational Software/Markku Tiainen
• Major problem area identified earlier (CBo): Software build process and related tools
• Major Tasks of this consultancy effort:
• Identify and isolate any problems further, analyze reasons and suggest and/or make it happen:
  • Short term remedial actions (within 4-6 weeks)
  • Longer term improvements (within > 6 weeks to 6 months)
  • Other considerations

Tasks, deliverables and schedule

• Follow up and reactivate remedial actions based on earlier reported technical problems for Rational Software Technical Support. Done.
• Interviews and basic analysis work during 14-15.12.1998. Done.
• Data collection and final interviews to check valid problem areas 12-13.1.1999. Done.
• Action plan draft and hand over to review by 19.1.1999. Done.
• Present and hand over this review 1.0 with suggestions by 1.2.1999. Done.
• Create a Project Plan*) including a realistic schedule to execute all the approved short term remedies actions or longer-term improvements. Will be done and reviewed during 1-5.2.1999.

Follow up and reactivate remedial actions based on earlier reported technical problems for Rational Software Technical Support.

Reported problems

Raised issues for Rational Software's ClearCase R&D

These issues have been raised during MGi’s visit to Rational Software in Lexington, MA 25-26.1.1999. All possibly agreed action points will be followed up by NTC and Rational Software jointly. Some of them have been solved, most of them are in progress and in detail reported by Marc Girod internally or via e-mail correspondence between him and Rational Software.

Statuses of different issues, (known by the time of writing this) and some new ones

Interviews and basic analysis work during 14-15.12.1998

General

• 7 personal interviews were performed within NMS For Data
• It was asked to raise any kind of problems, which has by experience a negative impact on productive R&D work for the person in question
• It was encouraged to suggest concrete actions how to improve or solve the raised problems
• NMS For Data has provided a good working environment and opportunity to to this review and all the participants have co-operated with 100% support. This shows a high level of commitment to improve both the whole R&D process

Problems expressed or observed and listed by category after all interviews:

HW & SW Infrastructure

• Server host availability and capacity is not enough for all developers
• Exceed emulator session not reliable enough
• Machine down-time during daily work hours can be up to 1-2 hours
• Users may have to come to work on some "low peak" hours instead of 8-16 daily.
• Loading and using Netscape Browser on Unix is very slow
• Computer roles and their available services are not clear enough

Analysis in short

Positive observations:

The organization is able to work and release software.

According to the users interviewed and based on some personal observations 1-2 days the above mentioned issues are true and they have been experienced from a longer period of time.

Negative impacts:

The fact that the HW & SW infra is not able to serve the organization well enough during the daily working hours should be considered critical, because it results in delaying productive R&D work for all developers and also demotivates people.

Short term remedial actions taken and decisions made (within 4-6 weeks)

• Make a plan how to upgrade HW & SW infra and include it as a sub-project plan to this main project plan *).
• Replace all existing HP/UX servers to more powerful and reliable RAM, disk space and CPU. Person responsible: Jukka Papunen from IT-Support.
• Analyze the LAN network in use including all participating HW & SW on the segments and fix all problems found. Person responsible: Jukka Papunen from IT-Support.
• After all computers have been replaced and their OS have been upgraded, upgrade existing ClearCase HP/UX versions to ClearCase HP/UX 3.2.1
• Upgrade also ClearCase MultiSite on HP/UX to use ClearCase 3.2.1 if the ClearCase MultiSite counterparts can do it in sync also. Avoid using mixed ClearCase environments such as ClearCase 3.01, 3.2. 3.2.1 together. In most cases they will work and are supported, but because newer releases introduce often also new functionality, this may lead to minor and annoying errors for the user. Check the compatibility of different ClearCase releases from:
1. ClearCase/MultiSite Release Notes, release UNIX 3.2.1
2. ClearCase Product Family Installation Notes, UNIX, release 3.2 and later
3. ClearCase V3 Migration Guide, Windows NT, UNIX, release 3.2 and later
4. ClearCase UNIX/Windows NT Interoperation, release 3.2 and later

Longer term improvements (within > 6 weeks to 6 months)

• Based on Services Required, walk thru the computer roles and reorganize, reinstall and update the needed HP/UX, other sw and tools used. Check all paths per OS (HP/UX 10.20 & HPUX 11.0 needed).
• The basic model to install ClearCase (either Unix or NT) is to have separate View and VOB servers or local View servers (clients). It is also recommended to application software on in other computers than VOB & View servers. Any other resource consuming applications e.g. Web-server should be installed and used in other than ClearCase server hosts.

Other considerations

• If performance becomes a bottleneck for ClearCase users after all performed infra changes, then a complete Performance Analysis consultive effort is recommended (e.g. from Rational Software)
• Check what new possibilities the ClearCase HP/UX Release 3.3 (3Q99) will introduce
• Check if it is feasible to start using NetApp (APPENDIX A) solutions with ClearCase on HP/UX.

SW Tools

• For developers having mainly NT experience, emacs-dependent environment is not easy to use or is considered not so productive

Analysis in short

Positive observations

The ones who have learned using the existing tools this is not an issue and for the whole environment servers well.

Negative impacts

For some designers and consultants (short-term contractors) who have had not chance to learn character-based tools, the ability to do productive work as motivated manner is not so good.

Short term remedial actions taken and decisions made (within 4-6 weeks)

• Encourage to use also available GUI based tools with ClearCase HP/UX clients

Longer term improvements (within > 6 weeks to 6 months)

• Encourage to use NT based design and programming tools where feasible and possible to do so. It is possible that application portability issues should be considered.
• Additionally install also ClearCase NT for those clients and designers, who are more willing to use it. Technically this has been possible and now there is an option to use ClearCase in a mixed environment with Syntax TotalNet NFS Server (APPENDIX B)using SMB protocol, which enables ClearCase NT Clients access ClearCase VOBs on HP/UX. (Before this was possible with Client side NFS products but with a little expense of network performance.)
• NB! Before starting to use NT clients and servers, make sure that there is a SysAdmin support person (and vice) available for NMS For Data organisation on demand and as needed.

Making an application release and build

• Slow to learn and execute
• An experienced user is needed often to help and complete the release
• The tools used are not integrated with the internal instructions. They are mostly in HTML format and available via Browser.
• Often the work effort goes to prepare and run a build only and it may take 3-5 days
• Config Specs are too complicated
• If 80 % of the week’s work time goes to do a build and 20 % is left for the designer’s main job, it would be better to chance vice versa: 20 % to do the build and 80 % should go to the main job effort: design / implementation of the application itself and releasing

Analysis in short

Positive observations:

The internal release instructions and helpful scripts in the Intranet are carefully maintained and accurate from the content point of view. Many advanced and powerful ClearCase features such as configuration record in build and labeling and wink-in are in use.

It is true that by following the internal release instructions via Browser Window and at the same time learning and doing the build with other tools was slow and difficult to understand. This would frustrate any developer until the tools and instructions are well understood and learned. However, after the learning curve achieved per designer, many developers considered this workflow not so problematic after all.

Negative impacts:

prevents or delays productive R&D application release work, does not motivate using the tools the intended way until they are all learned well enough.

Short term remedial actions (within 4-6 weeks)

• Arrange interactive training per subject such as:
  • walk-thru of the internal release instructions for all, 2 hours – 1/2 day
  • general makefile design and programming (using clearmake) for all, ½ - 1 days
  • arrange ClearCase Unix user level course, for inexperienced users, 2 days
• Instead of maintaining a single View for different tasks per user, start using them on a per task basis such as: New development, Bug Fixing, Releasing etc. The basic view "usage" pattern would be: create the View for the task, edit config spec, do the task in question, finish the task and remove the View. (The other positive impact would be the destroying of unnecessary View Private elements, which reserve disk space per View.)
• Raise the priority of the issue, update the project plan to include release practices and invest on making the release happen fluently with the current tools

Longer term improvements (within > 6 weeks to 6 months)

• Reorganize and simplify internal release and build instructions if possible, do not use and maintain a "link jungle" which may mix the developer more, consider their structural redesign
• Form a Release Team and make it responsible performing the major releases of application. The release team should include the following R&D Roles:

  Project Manager:

  as the team leader and having the whole responsibility of making the release and participating ClearCase activities such as labeling the source elements with the release labels and locking them for the application release.

  Tools Expert:

  who knows the main tools and release instructions well and is available for always during the application release to analyze and help fixing any minor or major related problems

  Designer:

  responsible for performing the release builds as planned and maintain the makefile environment per application

Other considerations

• Consider and start using ClearCase 4.1 and SUM when it comes available 1Q2000: Base level releasing + Change Sets using the graphical tools. This Usage Model is a powerful major feature in ClearCase HP/UX and NT platforms.

Other problems reported

• Customer deliveries are not in control and the designers are not used efficiently enough to share their knowledge in order to avoid further problems
• Customer may have a different environment than the in-house lab has, therefore it is not sure that customer delivery could be tested, configured and experienced as it should before delivering it to customer
• Work roles, responsibilities are not clear enough
• Testcases made by Cap Gemini J-P Riihelä are not in use
• Regression testing is not reliably in use
• Sub-contractors and consultants vs. NTC workforce maybe not be treated equally

APPENDIXES

NetApp

Statement from Rational Software about NetApp and ClearCase usage:

NetApp boxes support both NFS and SMB protocols, which means that technically NetApp servers can store data for both UNIX and NT clients. However, since Windows NT does not support links, only UNIX VOBs can have remote storage pools. You can however use a UNIX VOB, which has remote storage pools, from NT by specifying the location of the remote storage pool via the Split Pool Map field in the Region Synchronizer.

ClearCase - UNIX

NetApp file servers can be used to store UNIX VOB and view storage pools. ClearCase Servers and clients must be able to access the storage pools via a global pathname. This is typically done by creating a symlink from the pool directories on the ClearCase server to an NFS exported pathname on the NetApp File Server.

ClearCase VOB storage directories contain the following directories by default:

db

database files

s/sdft

source pool

d/ddft

derived object pool

c/cdft

cleartext pool

The storage pools are the only physical structures that can be moved to a File Server like NetApp. Generally, Network File Servers do not support running a native OS and therefore can not run ClearCase processes (albd, lock mgr, etc...). Since the ClearCase database servers run where the 'db' directory physically resides, the 'db' directory cannot reside on the NetApp Server. You can also store the '.s' subdirectory of view storage on a NetApp File Server.

NOTE: Although NetApp recommends using RAID level 5 in their White Paper regarding ClearCase and NetApp, Rational Software does NOT recommend this. Performance in RAID 5 configurations is generally too slow for VOB updates. Rational Software recommends running RAID 0+1 if possible and especially if server performance is limited.

ClearCase - NT

NetApp is in the process of doing some testing of an all NT configuration where the storage pool directories reside on their server and are accessed via SMB. Rational has no information to date about how well this does or doesn't work and therefore cannot recommend this configuration at this time. (1/99)

TotalNet from Syntax and ClearCase

http://www.rational.com/products/clearcase/dynamic.jtmpl

More details from related ClearCase release 3.2.1 manuals

More details available from Syntax:

http://www.syntax.com/whatsnew/company/rational53.htm