On Plug-ins and Extensible Architecture

In a world of increasingly complex computing requirements, we as software developers are continually searching for that ultimate, universal architecture that allows us to productively develop high-quality applications. This quest has led to the adoption of many new abstractions and tools. Some of the most promising recent developments are the new pure plug-in architectures.

What began as a callback mechanism to extend an application has become the very foundation of applications themselves. Plug-ins are no longer just add-ons to applications; today’s applications are made entirely of plug-ins. This field has matured quite a bit in the past few years, with significant contributions from a number of successful projects.

This article identifies some of the concepts around the foundation of pure plug-in architectures and how they affect various stakeholders when taking a plug-in approach. The intention is to discuss the general issues involved, but the article also presents some lessons learned from Eclipse (www.eclipse.org) to better illustrate some of the concepts.

MOTIVATION FOR PLUG-INS

Customers’ requirements control the creation and deployment of software. Customers demand more and better functionality, they want it tailored to their needs, and they want it “yesterday.” Very often, large shops prefer to develop their own in-house add-ons, or tweak and replace existing functions. Nobody wants to reinvent the wheel, but rather to integrate and build on existing work, by writing only the specialized code that differentiates them from their competition. Newer enterprise-class application suites consist of smaller stand-alone products that must be integrated to produce the expected higher-level functions and, at the same time, offer a consistent user experience. The ability to respond quickly to rapid changes in requirements, upgradeability, and support for integrating other vendors’ components at any time all create an additional push for flexible and extensible applications.

TRADITIONAL PLUG-INS VERSUS PURE PLUG-INS

Most people are familiar with traditional plug-ins, downloadable software bundles that extend the functionality of hosting applications such as Web browsers or text and graphical editors. They are not compiled into the application, but linked via well-defined interfaces and extension mechanisms. Most often, building a plug-in does not require access to the source code of the application. Plug-ins implement functions that the host application can recognize and activate when needed.

In the new pure plug-in architectures, everything is a plug-in. The role of the hosting application is reduced to a runtime engine for running plug-ins, with no inherent end-user functionality. Without a directive hosting application, what is left is a universe of federated plug-ins, all playing by the rules of engagement defined by the framework and/or by the plug-ins themselves.

To support composing a larger system that is not prestructured, or to extend it in ways you don’t foresee, the architecture must support the extensibility of plug-ins by plug-ins. In the absence of a hosting application, plug-ins themselves become hosts to other plug-ins by providing well-defined hook points where other plug-ins can add functionality. These hook points are commonly known as extension points. When a plug-in contributes an implementation for an extension point, we say that it adds an extension. Much like defining any contractual obligation, an extension model provides a structured way for plug-ins to describe the ways they can be extended and for client plug-ins to describe the extensions they supply.

In a simplified way, figure 1 illustrates the main structural difference between traditional and pure plug-in architectures.

Click for figure

PLUG-IN RUNTIME CONSIDERATIONS

As already mentioned, plug-ins are not stand-alone programs; their execution is orchestrated by a runtime engine. The plug-in runtime engine—let’s call it the kernel—is the core engine that is started when a user launches an application. At a minimum, the kernel must provide runtime support for the basic plug-in infrastructure by:

• Finding, loading, and running the right plug-in code.
• Maintaining a registry of installed plug-ins and the functions they provide.
• Managing the plug-in extension model and inter-plug-in dependencies.

Optionally, the kernel can supply other utility services, such as logging, tracing, or security. The user should also be able to turn off or remove parts of the application, or dynamically install new functions. For obvious reasons, it is desirable for the kernel to be small and simple, yet robust enough to build industrial-strength functionality on top of it, without many (or any) hacks.

To illustrate the basic workings of a plug-in environment, let’s take a behind-the-scenes look at the Eclipse plug-in runtime structure and control flow. While mostly known as a powerful Java integrated development environment, Eclipse is actually a universal plug-in architecture for creating “anything, but nothing in particular.” The runtime engine itself is implemented as a number of core plug-ins, except for a tiny bootstrap code. This code starts the core plug-ins to initialize the plug-in registry and the extension model and to resolve plug-in dependencies. Other than the core plug-ins, no other plug-in code is run at this time. All the needed plug-in metadata is read from the plug-in manifest files (plugin.xml and/or manifest.mf).

Once the initialization is complete, the runtime kernel is ready to run applications. The extension mechanism, shown in figure 2, kicks in and an entire application is built progressively, from inside out, as follows:

1. The runtime core plug-in defines an applications extension point so that any plug-in can declare itself an application by contributing an extension to it.
2. An applications extension must provide a concrete implementation for the extension point’s callback interface, IPlatformRunnable.
3. Eclipse’s default applications extension is contributed by the IDE plug-in, by providing a concrete implementation class that, when called by the extension point handling code, creates the workbench graphical user interface and runs the event loop until it exits.

Click for figure

This workbench window looks like any standard application: it has menus, toolbars, views, and editors, and you can launch wizards, open preference dialogs, and so on. The basic workbench by itself, however, is just a frame for displaying visual parts and other user interface elements. Its power comes from the extension points (e.g., views, wizards, editors, action sets, help support) it defines, to which other plug-ins, or the workbench plug-in itself, can contribute extensions.

In general, the plug-in that defines an extension point is responsible for looking up its extenders (using the extension registry), instantiating the associated callback class, and invoking the appropriate interface methods. Under the covers, the runtime kernel is responsible for providing the extension registry lookup functionality, creating a class loader for the extender plug-in, loading it, and starting it. This ensures the extender plug-in is initialized and active before its classes are created and run.

Finally, it is worth noting that some extension points may require more than one callback interface and others do not require any. For example, the help system extension point toc is used for contributing documentation tables of contents and does not require any callbacks. When help is invoked, the help plug-in processes the toc extensions by creating a larger, integrated table of contents from all contributing plug-ins.

CONFIGURING AND DISCOVERING PLUG-INS

An important consideration when deploying a plug-in system is configuring and discovering the plug-ins. Unlike monolithic applications that are typically installed in one folder on a user’s machine, plug-in-based applications may have a higher degree of freedom for installation layout and plug-in discovery. This flexibility can also be a source of major headaches for installers and plug-in management.

Consider a multiuser shared installation where the entire product is installed by the central administrator on behalf of a community of users. The individual users do not need to perform any installation. They are simply given a local “shortcut” to invoke the shared installation (the shortcut can be in the form of a command-line invocation or a true desktop shortcut encapsulating the command-line invocation). If you’re like most users, you will want to install some other cool new features, especially when the third-party plug-in offerings are plentiful (and often free). Obviously, the new plug-ins, private to that user, cannot be installed in the read-only, shared install location, so the product should allow users to install and configure extra plug-ins in a location where they have more privileges.

Configuring only a subset of plug-ins for a particular user based on access criteria, such as user roles, adds more challenges. Other configuration issues arise in scenarios where a common set of plug-ins is shared by many applications running on the same machine, or where some plug-ins are not installed locally but run from a remote location (e.g., an application server provider).

In theory, the platform must find the available plug-ins on its own and gracefully cope with missing plug-ins or those that are dynamically coming and going. In practice, many solutions for these problems involve predefined files and directory locations for key configuration files or for the plug-ins themselves. More advanced plug-in systems offer pluggable configuration and plug-in discovery mechanisms. Usually some basic bootstrap code is run to start a configurator plug-in, which will then discover the other plug-ins based on its own strategy. For example, Eclipse provides an Update Manager configurator plug-in that picks up plug-ins from the eclipse/plugins folder, as well as from other local plug-in folders linked from the eclipse/links folder or dynamically added when users install new plug-ins to locations of their choice. Because the configurator is pluggable, anyone can plug in another configurator, which can provision and configure the plug-ins from a remote server.

CHALLENGING ISSUES

Pressured by time, budgets, or slick marketing claims, you may be tempted to adopt a pure plug-in architecture without being aware of its potential pitfalls. Alternatively, you might truly believe in Murphy’s law and therefore won’t use plug-ins because they’re doomed to fail, anyway. I tend to follow those who walk the middle road: if something can fail, then it first should be understood and then fixed. The rest of the article presents issues that are likely to pose some challenges when you employ plug-ins, and that will help you ask questions or provide answers when evaluating a plug-in architecture.

Installing and updating.Many modern products automatically detect when they are out of date with respect to available service or product version. Either on start-up, or as a result of an explicit update action, the products compare the current installation level against some network-based baseline. The product then automatically downloads required fixes or upgrades and applies them, often as part of the product execution. Additionally, users can, and most often will, install additional plug-ins from various sources to extend the functionality provided by their current application.

For plug-in-based applications, the installation and update process can be a real nightmare: on the one hand, there are the traditional installation issues that arise in any application—ability to roll back changes, migrate existing program data and preferences, or ensure the installation is not corrupted. On the other hand, because plug-ins may originate from various providers that are not related to each other, the resulting configuration has likely never been tested. This poses a number of interesting challenges that we address in the context of the other issues discussed in the next sections.

Security.Systems can never be too secure, and you need to pay particular attention to securing a system based on plug-ins. Since arbitrary plug-ins can be installed—for example, by downloading them from the Web—and are allowed unlimited access to the system they plug into, security in a plug-in environment must be carefully planned. On top of this, some plug-ins require support for executing custom install code during installation, so they can have control over some parts of their installation. To prevent software security accidents or failures, the plug-in framework must address the issues of downloading from third parties and controlling a plug-in’s access to other code and data. Supporting digitally signed plug-ins or secure connections helps, but it still relies on trusting the download source or the plug-in provider. Some programming environments, such as Java, offer built-in runtime security mechanisms that can be used effectively to close some of the security gaps.

The cold reality is that, unless you are careful about what you install, it is almost impossible to be confident that the installed plug-ins are not ill-intentioned. Of course, a pure plug-in architecture also means that even a well-intentioned plug-in with serious bugs can do as much damage as an ill-intentioned one when installed.

Concurrent plug-in version support. Without a doubt, managing concurrent plug-in versions and dependencies is one of those problems that can keep architects, developers, and installation folks awake at night. Most of you have probably experienced “DLL hell” at some point and will look with suspicion on something that has the potential of being a “plug-in hell.”

Any serious plug-in model defines some versioning scheme, but allowing multiple versions of a plug-in to be concurrently installed and executed is an issue that must be considered early on in the design, and it must be properly enforced by the install and update processes.

To illustrate the complexity of this area, let’s start by considering the scenario of an application suite that integrates two stand-alone plug-in products, each installed in a separate location. It is possible to have the same plug-in, at the same version or at different versions, installed in two locations. The runtime kernel must deal with the two plug-in instances by either running them both or dropping one. When both versions are running, and the plug-ins contribute user interface elements such as menus or preference pages, the result may be a very confusing interface, with duplicate menu entries or preference pages. There is more trouble: would the user need to be exposed to this anomaly? How would you update or uninstall these plug-ins?

The difficulty of the problem is amplified by the other important role plug-in versions play here: they are part of plug-in dependency specifications. Typically, a plug-in requires the functionality provided by other plug-ins, and very often there are strict criteria about what versions are required. For example, a plug-in may require a certain level of an XML parser contributed by another plug-in, or it may need to contribute an extension to a particular version of an extension point only. Version dependency can be in the form of a fixed version number such as 3.0.1, or as a range of versions—say, 3.0.1 or newer. Properly resolving plug-in dependencies at runtime is critical to the correct functioning of the application. Managing plug-in dependency graphs is also critical to ensure a consistent configuration state after installing or updating plug-ins, as well as for properly rolling back changes. The install/update process may need to abort installation of plug-ins that leads to an unresolved dependency, or search for and install the missing plug-ins. Either way, it is possible to end up with multiple versions of the same plug-in, so we’re back to the problem of managing concurrent versions.

There is no simple, general solution to managing concurrent versions of plug-ins. Eclipse has adopted a reasonable trade-off convention for concurrent plug-in versions: only versions of plug-ins that contribute code libraries but no plug-in extensions (no user interface contributions, no documentation, and so on) are allowed to coexist in the same runtime instance. For all the other plug-ins, the latest version is usually picked up, unless a configuration file precisely defines what to run. The main advantage of this approach is that it still allows various levels of the same code to coexist at runtime, but hidden to the end users, who will get a consistent user interface. The downside is that this will not cover all user scenarios, and that plug-ins are not treated uniformly as elsewhere in Eclipse. This special treatment of plug-ins is not just a runtime/install issue, but also something that developers and product packagers must consider.

Scalability, up and down. Another challenge of working with plug-in architectures is scalability. Just as the problems associated with multiple versions and dependencies can quickly escalate, so too can the sheer number of interacting plug-ins quickly become a problem. For example, when Eclipse was first designed, it was thought that a product, a large one for that matter, would consist of a few hundred plug-ins. A few releases later, some enterprise-class products built on Eclipse are known to have passed the thousand plug-in mark, so the platform goal has been revisited to support scaling between 5,000 and10,000 plug-ins.

When designing a plug-in system for scalability, developers must consider various mechanisms that make start-up faster and have a smaller memory footprint. A general principle of runtime is that the end user should not pay a memory or performance penalty for plug-ins that are installed but not used. A plug-in can be installed and added to the registry, but the plug-in will not be activated unless a function provided by the plug-in has been requested according to the user’s activity. In general, this requires support for plug-in declarative functionality. It is often realized in practice via plug-in manifest files, so no code has to be loaded for obtaining the function contributed by plug-ins. Caching of the plug-in registry and the plug-in manifests/declarations can reduce processing during start-up in subsequent application launches, improving response time. What is gained in performance and memory footprint, however, is lost in code complexity: more code needs to be written to cache the data and to synchronize the cache with changes in the installation configuration.

Scalability problems almost always surface during plug-in install/update operations. The larger the product, the larger the download size of patches and upgrades. For product upgrades, the download time can be improved by simply downloading only the plug-ins that changed. Another installation scalability problem can arise during an interactive install/update operation. Quite often plug-in boundaries are established for development reasons (such as function reuse) and present the wrong level of granularity in terms of what the user sees as the unit of function, and therefore as an installation unit. Considering that enterprise-level applications can scale up to hundreds and thousands of plug-ins, with complex dependencies, the user can be overwhelmed by various installation choices. A possible solution is to introduce a packaging and installation component that groups a number of plug-ins to offer a higher level of function. For example, in Eclipse, Update Manager does not install plug-ins directly; it processes features. Features are bundles of plug-ins and are considered deployment units with install/update semantics.

The other side of scalability is scaling down, so products can run on devices with limited resources, such as cellphones and PDAs. This usually means rethinking the core framework plug-ins to refactor them into smaller plug-ins, some of them optionally deployable, so that one can run with a minimal configuration.

OVERCOMING THE BUMPS

General discussion about plug-in architecture and design issues may not mean much for you or your company’s bottom line unless you’re a participant or plan to be one—either by creating a plug-in framework or developing plug-ins for fun and profit, or by using a plug-in-based application. Employing a plug-in framework is not without bumps, but they can be overcome with proper preparation.

While traditional plug-ins have been around for some time, the pure plug-in models have only recently emerged as robust, enterprise-level quality application development environments. As many major industry players are rapidly adopting plug-in technologies for their software lines, we can expect further research and development into improving the current architectures. Some of the areas that will receive increased focus will be:

• Security at all levels, including installation, update, and runtime.

• Performance: speed and low resource usage.

• Improved tools for development, testing, packaging, and deployment of plug-ins.

• Improved installation and updates, especially by merging changes without taking the system down.

• Remote management of various processes (provisioning, configuration, and so on).

• Convergence, compatibility, or interoperability of various plug-in frameworks.

• Wider range of deployment platforms (from desktop PCs and high-end servers to digital mobile phones and embedded devices).

Increasingly, much of this effort will be happening in the open source domain or standards bodies.

Back to On Plug-ins and Extensible Architectures

http://www.cer.net2003-04-07 11:36人民网

作者：韩冰

    眼下读MBA，到底选洋的还是土的？北大国际MBA招生办主任张黎说，如果申请人希望未来成为“国际人”，经济实力又比较强，首选当然还是美国和欧洲的名 校，像美国《商业周刊》评选的商学院。美国的商学院已经有百年历史，其办学质量、师资、学生的素质和学院的管理都不是国内商学院现在就能比的。但国内商学 院的机会已经来了，从2000年起受经济影响，美国商学院学生的就业情况不好，国际学生就业就更差，连哈佛商学院毕业的国际学生在美都难寻工作。据可靠消 息，目前美国的顶级商学院，像芝加哥大学、沃顿商学院对国际学生的名额和助学金都做了严格限制，目的是减少国际学生，保证本国学生的就业，以保住在《商业 周刊》等排行榜上的地位。这也与今年哈佛商学院的中国毕业生全部回国，芝加哥大学等些一流商学院的一批中国毕业生回国就业互为例证。

    从投入产出比看，上哈佛共需花费10万美元。二流商学院也要6～7万美元，在美国的中国学生一般都会贷款求学。而国际学生毕业后就业面窄，还受到还贷的压力，导致他们不去生产型企业，只能去跨国公司中的咨询公司和金融公司，这些公司目前提供的职位根本不能满足需求。

    如果选择回国创业，就需要对中国文化、风土人情都很了解。中国经济发展如此迅速，在国外2～3年，回国后很难马上适应。所以创业这条路是艰难的路。

    如果选择国内商学院，常规MBA一般需花费10～12万元，合1.5万美元。职班的人也不用辞职，利用业余时间上课就行，投入就更小。从就业看，国产 MBA虽然在跨国公司的职位竞争上不占优势，但在民营、国企大公司的就业机会就很多。另外，跨国公司人才本土化的过程也推动了国产MBA的就业。

    如果选择国内商学院，当然选一流的为好。目前中欧国际工商学院、北大国际MBA、北大光华管理学院、清华经管学院、长江商学院都是公认的一流商学院，各有特点。

    作为申请人，可从三方面选择：

    一是选择什么样的学生群体(是来自外企圈还是来自国企圈)；

    二是选择什么样的老师(是来自美国还是来自欧洲)；

    三是课程(是否有学习方向，是否有本土案例)。建议申请人报名前先到自己中意的商学院进行实地考察，到课堂试听之后再决定。

http://www.cer.net2003-01-03 14:06新华网

作者：姜帆 王攀

　　要读MBA，对不同需求的考生来说，选择也应是不一样的。目前，在国内读MBA，有5种可供选择的模式。

　　第一种：参加全国联考，达到教育部和报考学校的分数线后入学。毕业颁发教育部承认的硕士学位和学历证书。这种考试要考高等数学和线性代数，考生 比较辛苦，但学费相对低廉，对工作经验的要求不是很严格。这种类型一般还分为春季班(脱产)、秋季班(在职)和经贸班(有学历无学位)等几种。

　　第二种：各个学校组织的培训性质的工商管理教育，入学不需要经过全国联考，学费较高，但形式多样。包括EMBA(高级管理人员班)、EDP(高 层经理培训课程)、MBA课程研修班、短期培训班、论坛(以报告会形式展开)等。后4种都没有考试，不需推荐，没有学位，对入学者工作经验的要求不严。

　　第三种：中外合作办学，毕业后发教育部认可的国外商学院的学位和文凭。北大、清华、复旦、南京大学、中山大学等都有此种形式，其中最有名的是北 大中国经济研究中心和美国26所商学院合办的项目。特点是英语和工作经验要求较高，不考数学，可脱产或在职学习。理论上说，学位在中美两地都有效，但学费 之高令人咋舌，而且还是美元。

　　第四种：来自世界各地的大学来华办的从学士(BBA)到博士(DBA)的各种工商管理的教育课程。有名校的，也有一般的学校，但无一例外，国家都不承认其学历学位。

　　第五种：中欧工商管理学院。该校的办学质量有很好的口碑，毕业生很受企业青睐，遗憾的是学费非常昂贵，国家还不承认学位。有人戏称：“中欧是那些英语足够好，钱足够多，但又去不了哈佛的人的最好选择。”

　　参加全国联考，是最为普遍的一种模式。根据教育部的规定，MBA联考报考人员必须是：研究生毕业后有2年以上工作经验者；大学本科毕业后有3年 以上工作经验者；大专毕业后有5年以上工作经验者；年龄一般不超过40周岁。各招生院校可在教育部规定的报考条件基础上规定本校报考条件。MBA联考的科 目为：英语、数学、管理、语文与逻辑、政治。前4科为统一命题，每年上半年教育部MBA联考指导委员会公布次年考试大纲及核定教材，政治理论课由各招生院 校单独命题、批卷。联考各科的内容每年都有所变化，总趋势是向实践性，综合性，灵活性的方向倾斜。死记硬背的概念、定义类的内容会逐渐减少，而着重点考察 考生的理解能力、综合分析解决问题能力。

　　联考分数达到报考院校的分数线后，再通过录取评审委员会对申请材料的评价以及差额面试，考生才能入学。

　　一些专业人士认为，目前的联考制度有待改进。在联考中考了高分的学生未必就有管理能力，而相当一部分有管理经验和管理潜力的学生因为在分数线以 下就只能和MBA擦肩而过。南京大学MBA中心的桑秉南主任说：“既要兼顾社会公平，又要选拔具备管理才能这种不可量化的资质的人才，MBA联考就成了两 难选择下不理想但不得不为的解决方法。”

Discussion

Overview

The cost impact to a company of a failed project can be severe indeed. The impact on the reputation of the project manager can be disastrous.

Software project management is not easy, and it requires considerable skill to successfully manage the many different risks that conspire to de-rail a project:

Numerous methodologies are available for mitigating these risks – PRINCE2, RUP, DSDN, eXtreme programming – and these have helped to some extent.

This document introduces the 3D™ methodology – a set of best practices and quality tools developed by BuildMonkey, which can be summarised as.

De-risk. Deploy. Deliver.

In any software project, deployment is a milestone on the project plan that is usually associated with payment – or staged payment. Through the course of development, any number of problems can arise to derail efforts to reach this milestone.

The 3D™ methodology and supporting toolset is based on many years of experience at the sharp end of projects, understanding what has worked and what has not, and the lessons learned from each.

Competent practitioners, and experienced project staff, will find resonance with many of the contents of this document and may find themselves saying “this is just common sense”. This is certainly true, but the main problem with common sense is that it is not as common as people think it is.

This document, and the 3D™ methodology, is an attempt to bring that common sense together in a single location, as a coherent set of best practices supported by proven tools to help you to release on-time, on-budget, with no defects.

The Problem To Be Solved

No methodology has yet focused on the component that all development projects share – the build.

One of the reasons for this is that the term “build” is interpreted differently by different people:

• The development team sees it as compilation and assembly;
• The integration team see it as the bringing together of all of the components in the application in a format suitable for release;
• The deployment team see it as something which produces the artifacts that they have to install and configure;
• The testing team see it as something which produces the artifacts that they have to test;
• The Project Manager sees it as an opaque step that nobody is entirely responsible for;;
• The end customer should not see it at all;

The BuildMonkey view is that the build is the combination of processes and technology that take software from design to deployment – _where the return on investment starts to be seen.

It is clear that a methodology is required to de-risk development projects and to standardise use of the term “Build Management”.

Best Practice: “Build Management” encompasses everything from compilation, all the way through to release to the customer.

No Release, No Revenue

Any Finance Director knows that development is just an activity that needs to be tolerated in order to produce something that will deliver a return on investment.

It may sound strange, but a large number of software developers do not appreciate and embrace this basic truth. This is in part due to their closeness to the application being constructed.

A common problem faced by development projects is therefore that it is the software developers who manage the build. This creates a situation where the build is focused on the needs of development, and is not geared towards releasing the output of coding such that business value can be realised.

Build Management should therefore focus on the end result of development – a return on investment – and ensure that all of the inputs to the process are incorporated in pursuit of this goal:

Best Practice: Focus on the end, and accommodate the inputs

Software Tools Are Only Part of the Answer

Software projects are a complex set of interdependent people and teams and can be likened to a convoy of ships. A convoy has to move at the rate of the slowest ship. Increasing the speed of a single ship in the convoy will not increase the speed of the convoy – it will simply increase the amount of wasted capacity in the speeded-up ship.

Speeding up the slowest ship will, however, have a positive effect since the whole convoy can now move faster.

Many Project Managers try to improve productivity by implementing some degree of automation in development projects – particularly in the area of the build – and often purchase “magic bullet” build software that provides this.

Simply using automated build software does not improve productivity any more than the example above improves convoy speed - as it only increases the speed of a single ship in the convoy.

There is no point in speeding up development, if the target production infrastructure cannot keep pace – this just increases the inefficiency. A lot of organisations make this mistake – highly agile development processes trying to feed into considerably less agile deployment processes. The result is inefficiency, waste and over-run.

Before considering using an automated build tool it is essential to ensure that the inputs to, and outputs from, the build can cope with the improved build speed. It is imperative to ensure that the processes and technology employed are geared towards taking the project to a successful conclusion – on-time and on-budget.

Best Practice: Don’t rely on software tools alone, they may solve symptoms whilst creating problems elsewhere

Configuration Management Best Practices

Software Configuration Management (SCM) is a relatively mature discipline with much written about methodologies and techniques, and these will not be recreated here.

We will focus instead on leveraging the SCM repository, and the facilities that it offers, to further the goals of the project rather than to consider SCM in its own right.

It All Starts Here

The SCM repository - how it is managed and used – is the keystone of good build management and successful delivery of projects.

The SCM repository is the slave of the project, not the other way round. It should be solid and reliable, yet flexible enough to accommodate the needs of new projects. Project Managers should not have to retrofit their planning to accommodate an inflexible SCM setup.

If used correctly, the SCM repository will enhance productivity, and minimize risk, through being able to provide exactly what the project – and project management – require. If used incorrectly, it can cause delay and slippage through having to do things inefficiently further down the chain.

Best Practice: The SCM repository is the slave of the project, not the other way round.

Configuration Management is not Version Control

Most software developers regard the SCM repository as a massive storage area where they simply check versions in and out – a common cause of problems.

Simply checking things into an SCM repository is not Configuration Management any more than karaoke is opera.

A well-maintained SCM repository is so much more than version control, and should provide:

• The ability to recreate any identified baseline, at any time;
• Meaningful statistics on what is changing, when and by whom;
• Management information, such as “how many new defects were introduced by the refactoring of component ‘X’?”

In order to be truly effective in a project, the SCM repository should store all of the artifacts that form part of a baseline or a release.

Most development projects simply store the code that is being developed and their use of the SCM repository is no more sophisticated than this.

Most applications nowadays are not just source code. Take the example of a modern computer game – the vast majority of the code base is made up of artifacts other than code such as audio clips, pictures and movie clips.

Where an application relies on a database this database will have a schema and structure that may change from release to release – this schema must be captured.

There will normally also be seed data for the database which should be considered as part of the baseline.

In a large distributed application, the software being developed will sit atop a number of pieces of software (e.g. application servers, web servers and message queues).

The configuration of these underlying applications have an effect on the quality – or otherwise – of the software being developed and should, therefore, be considered part of the baseline for a release.

The underlying environment and infrastructure is a key component of the project, particularly in the case of distributed applications.

Such banal considerations as DNS zone files, user account information and system parameters have to be considered as some of the moving parts which affect the application and therefore be placed under configuration control.

This is of particular importance when there is more than one environment involved in the project (e.g. a development environment and a separate test environment) since the question of “are these environments the same?” crops up again and again.

Best Practice: Everything that can be changed, and affect the behaviour of the application, is a candidate for configuration control

The Point of Labelling

It is a common misconception that simply applying labels, or tags, to the SCM repository creates a baseline but this is only partly true without corresponding records of:

• What label has been applied;
• When that label has been applied;
• Why it has been applied (i.e. what milestone, or other external event, the label is associated with);

The use of a naming convention for labels can deceive even further. For example, a project that uses a date-based labeling convention (dd_mm_yyyy) will have several labels of the form (03_05_2004, or 09_06_2004) and will reasonably assume that they have some kind of record of the baseline on those dates.

But what was happening in the project on those dates? Was the 03_05_2004 label applied immediately before a release to test, or immediately after?

Best Practice: Labels should be used to identify and inform about events in the project

This may seem like stating the obvious, but there should be a reason for a label being applied – the whole purpose of labeling is to identify some event in the development cycle that may need to be re-visited.

To this end, labels can be divided into two categories:

• Point-in-time
  Associates particular versions with a particular calendar date, or other event that is fixed in time (e.g. MONTH_END_JAN_2004, or RELEASE_1_0_1);
• Point-in-process
  Associates particular versions with events in the project that may recur at a later stage (e.g. LATEST_RELEASE, or CURRENTLY_IN_UAT);

Best Practice: Every label should have a point, whether point-in-time or point-in-process

Management Information

The job of the Project Manager, ultimately, is to bring the project to a successful conclusion. If this were an easy task that happened by default, then there would be no need for a Project Manager.

In order to be able to do this job well, a Project Manager needs information. He needs to know what is going on in the project – who is doing what, who is working on which components, and a wealth of information can be obtained from a well-managed SCM repository:

• What is changing in the environment – what has changed since a given point in time or how often particular elements are changing;
• Who is changing things in the environment;
• Why things are changing in the environment;

Of course, the final item in the list requires that committers are using descriptive comments to indicate why they are marking a particular change. A well-managed SCM repository should enforce this.

Best Practice: The SCM repository should provide meaningful, and accessible, management information

Build Best Practices

As explained at the beginning of this document, the term “build” means different things to different people. The most common interpretation is the one used by developers, where the term “build” describes the compilation and assembly step of their development activities but this narrow interpretation is a common cause of problems and over-runs, on development projects.

Building is not Compiling

At the outset of the project, the Project Manager will ask the question “how long to set up the build?” and a developer – thinking of compilation and assembly – will answer something like “1 day” – a task and duration which is then duly marked on the project plan and development begins.

Later in the project, when it is time to start deploying and testing the application, this “build” needs to be refactored to accommodate the deployment and test tasks. In doing so, it turns out that the way the application is being assembled is not conducive to it being deployed or tested correctly – so the compilation and assembly staged need to be refactored as well.

In the meantime, the development team sits on its hands whilst the “build” is refactored to accommodate the needs of the project – valuable time is lost whilst the deadline continues to advance.

Best Practice: Know what will be required of the build before starting to write the scripts

Don’t Throw Out The Baby With The Bathwater

From a build perspective, projects with similar architecture (both in terms of the team and the application) will have similar attributes. There will obviously be some changes required, but these will tend to follow the 80/20 rule to a large degree.

For example, a web application that is being developed by an in-house team and that will be deployed to a Tomcat servlet container and Oracle database will follow largely the same steps and require largely the same deployable artifacts.

A good SCM repository will enable the latest versions of boiler-plate build scripts for such an application to be found. These can be used almost off-the-shelf – meaning that development can start apace without having to wait on the build to be constructed for similar applications .

Best Practice: Well-crafted builds are re-usable and should be re-used

The Architecture Drives The Build<

Following on from the previous section, it should be clear that the architecture of what is being developed – and the structure of the team(s) developing it – will dictate how the build should look.

There is little value to be gained in trying to retrofit build scripts for a computer game (developed by 12 people all in the same room) into a project to produce a large J2EE application with development occurring at six different sites around the world.

Best Practice: Well-crafted builds are flexible, but a “one-size-fits-all” approach can be costly

Management Information

There are a number of people who need information that the build can provide:

• The Project Manager needs to track overall progress against defined milestones – number of outstanding defects, whether a committed release date will be met etc;
• Development leads need to be sure that the code is of the quality that they require – test reports, bug analysis patterns, code metrics, document and UML generation etc;
• The deployment team need to know that the artifacts will work in their target environment, and that the environment is as the application expects it to be. They also need to know whether two (or more) environments are “the same”;
• The test team need to have confidence that they are testing against a known baseline, and whether defects that they see have been rectified in development (or whether they are re-appearing after already being fixed);
• Everybody needs to be able to communicate effectively using the same language, and have a common terminology for release versions – particularly if there are multiple threads of development;

A good build infrastructure will provide all of the above information, and more besides.

Best Practice: The build should tell all project participants what they need to know

Deployment Best Practices

Considering that it is generally an important milestone on a project plan, normally resulting in payment or a staged payment, deployment is one of the most overlooked areas of software development.

The normal course of events is:

1. Release artifacts are created;
2. Some installation and release notes are cobbled together in the form of a README;
3. The deployment team work frantically to install and configure the application – the testing team (or, worse still, the customer) are idle and unproductive in the meantime;
4. Some symptoms are found which are suspected to be application defects;
5. The development team blame the environment;
6. The deployment team blame the application;
7. Repeat (5) and (6) ad nauseam.

When a documentation team are also considered - responsible for creating documentation that the end user will need to install, configure and use the application – the situation becomes even more difficult.

This situation can be avoided by planning for deployment from the beginning. Deployment is an inevitable part of software development, yet it always seems to take people by surprise.

Best Practice: Know that deployment is inevitable, and incorporate it into the automated processes

Deployment Is Not Installation

As part of normal development activities, artifacts are installed into sandbox environments – and test environments – many times. But this is not deployment, this is installation.

In order to get an application into its production environment, be that an internal environment or on hosted-infrastructure, a number of hurdles must be overcome:

• The application must pass UAT;
• The application must be installed and configured correctly:
• All pre-requisites for the application must be satisfied;
• The end customer must accept the handover;

Deployment is that point in the life of an application where it starts to produce a return on investment. “Launch”, “Go-live”, “Release”, “First Customer Shipment” are all phrases which describe the same event.

Best Practice: Deployment is the point where an application starts to provide a return on the development investment.

The Environment Is a Refactorable Component

This point cannot be stressed enough, particularly in large distributed applications.

Every application, large or small, has a runtime environment in which it operates. In a simple desktop application, this is a standalone machine (such as a PC). In larger applications, this will be a combination of machines (e.g. an application server and a database) operating together to provide the runtime environment.

In either case, the application expects certain facilities to be available from the runtime environment and will function incorrectly – or cease to function – if these are not present .

The environment itself, whether standalone or a network, contains many moving parts that can be independently configured. IP addresses, or hostnames, can be changed. User privileges can be modified or revoked. Files and directories can be removed. Each of these can have an effect on the way that the application behaves.

In an environment that is owned and managed internally this can be bad enough. In an environment that is owned and managed by an external third party, and where project success is contingent upon successful UAT in that environment, this can be disastrous.

Best Practice: Be able to identify whether the deployment environment is as prescribed, and “fit for deployment”

One of the most common questions that arises in development projects containing more than one environment is, simply, “are these environments the same?” and its answer can be elusive.

It is essential to be able to answer that question – quickly and accurately – so that any perceived defects in the application can be categorised as “defect” or “environmental”.

This ability becomes particularly poignant where on or more of the environments are owned by different teams, or organisations.

Best Practice: Be able to prescribe what the deployment environment should look like, and have a capability to test it quickly.

The environment, as explained earlier, is a refactorable component. It can be changed, and parts can be moved or deleted. However, unlike application code, changes may need to be made to the environment in response to external events (e.g. hardware failure, or security policies).

Applications, particularly complex ones, use regression tests to ensure that observed behaviour after a change is exactly as it was before the change was made. The same should be true of the environment.

Best Practice: Automated regression tests for the environment that will compare observed behaviour both before and after changes are made.

For example, suppose that a number of operating system patches or service packs are applied to an environment where the application has been, or will be, deployed. How are these tested? Do you wait for users, or testers, to start calling to say that there are problems?

Or do you make sure that you know what problems have been introduced before your users do?

As stated earlier, the SCM repository should be used to store any artifact that can be changed and that may have an effect on the environment.

It may not seem obvious, but some of the most obscure environmental changes can cause an application to fail:

• Hostname resolution;
• Non-existent user or group accounts;
• IP and network connectivity;
• Existence, or otherwise, of files and directories ;
• Application or operating system configuration files;

It is essential that these environmental variables be placed under configuration control and able to be identified as part of a baseline.

Best Practice: Environmental artifacts that are not part of the application should be part of the baseline

Automate, Automate, Automate

Every single task that is performed as part of a development project – throughout the entire lifecycle – can be placed into one of two categories:

1. Tasks which require some form of human judgment;
2. Tasks which do not;

Tasks which fall into the first category can use some degree of automation, but should stop and wait for human intervention wherever judgment is required.

Tasks in the second category should be automated. There is no value in having expensive resources employed to do mechanical or repetitive tasks that a computer could do more quickly, accurately and consistently.

Best Practice: Automate anything that does not require human judgment

A note of caution - it may be tempting to think that automation will increase productivity on its own, but this is not necessarily the case. Automating an inefficient process will simply magnify its inefficiency – as explained in the section on Software Tools are Only Part of the Answer.

This, and it is worth repeating, is a common error – to assume that automated build software alone will improve productivity.

Best Practice: Do not automate inefficient processes, or you will only maximize the inefficiency

About BuildMonkey

BuildMonkey is a division of Nemean Technology Limited - a leading technical innovator specialising in Agile Deployment and build management.

We have over a decade of experience in helping clients increase productivity in their build, integration and test cycles. We have a track record of massively reducing costs and defects through process automation and proven methodologies.

Formed in 1999, and profitable every quarter, we invented the discipline of BuildMaster - we are the original and the best.

We provide specialist Build Management products and services - when all you do is Build Management, you get pretty good at Build Management. Our world-class Professional Services staff are the best in their field, complemented by original software products of the finest quality.

We aim to be the leading provider of Build Management products and services in the UK by providing excellent service to our customers, and by empowering Software Project Managers to aim for release on-time, on-budget, with no defects.

About the Author

John Birtley has been a specialist in Configuration and Build Management since 1994 with a focus on deployment - particularly the packaging and installation of developed applications. He has worked for a number of blue-chip organisations including Sun Microsystems, Hewlett Packard, Sony and Ericsson in the transition from development to deployment. BuildMonkey was launched in January 2004, to package and commoditise this cumulative experience under the title of "Agile Deployment". He lives in Southampton, where dogs and children conspire to take what little spare time he has.

PRINTER FRIENDLY VERSION

Flier Lu问到CLR Loader和Java Class Loader有什么不一样。要回答这个问题不容易，因为我对Java一窍不通。但既然问题提出来了，打肿脸来充胖子也得回答啊。

于是今天下午我在google上逛了一圈，找了些关于Java Class Loader的文章看了看。我的结论是，和Java Class Loader类似的东东，在CLR里是不存在的。

简单的说，CLR里不存在一个类似于Java.Lang.ClassLoader的东西。所以你无法实现你自己的CLR Loader。CLR里只有一个Loader，那就是CLR Loader。当然，CLR提供了自己独特的方式让你做动态装载。你所拥有的自由度是远比Java要大得多。

下面是具体分析。

我对Java Class Loader的理解主要是从这两篇文章里来的。

“Inside Class Loader” by Andeas Schaefer (http://www.onjava.com/lpt/a/4337) 和 “The basics of Java Class Loaders” by Chuck McManis (http://www.javaworld.com/javaworld/jw-10-1996/jw-10-indepth_p.html)。我看了看JVM Spec。但是我觉得不如上面两篇文章清楚。

先说Java里的Type吧。Java里的Type就是Class 。Class就是Namespace + class name.  Class通过Class Loader来装载的。System Class Loader缺省的话只在CLASSPATH中寻找Class。如果你要在CLASSPATH之外转载Class的话，你就需要自己的Class Loader。如果两个Class有相同的名字，并且在同一个Class Loader里，那么它们就被JVM认为是相同的，可以互相赋值。如果有一个不一样的话，它们就被认为是不一样的。互相赋值会发生ClassCaseException。换句话说，Class name + Class Loader是一个Type的独特的ID。

Java的这个模型有很多问题。首先它把Class name做为Type的ID。两个Class如果有同样的名字，但是实际内容不一样的话，如果它们被同一个Class Loader装载，JVM会认为它们是同一个 Type。这个太搞笑了。结果就是Class Loader必须要有一些特别的办法来管理Class name。一般的办法是加一个前缀，比如说Class的URL。如果Namespace管理不好的话，很容易就是安全漏洞。这是为什么JVM Spec里提到Class Loader必须要让Parent Class Loader先搜索Class，再自己寻找。而且Class loader必须要保存Class resolution的结果，这样下次Class resolution的时候，Class loader会返回同样的结果。Java Class Loader有太多的限制，同时又有太多的责任。

另一个问题是共享问题。如果同一个Class被两个不同的Class Loader装载的话，JVM认为这两个Class不是同一个Type，不能互相赋值。结果就是Class无法共享。被不同Class Loader装载的Class无法直接对话。直接对话会产生linkage错误。它们只能间接对话，比如通过Interface，或者共同的Base Class。

还有一个问题是Versioning。因为Java的Type里只有名字，所以当你看到一个Type的时候，你无法知道它是第一版，还是第二版。所以如果你需要第二版，但是Class loader给你装载了第一版的时候，祈祷吧。也许上帝能救你。

现在回过头来看看CLR的模型。CLR的Type包括两部分：Class name + Assembly Name。Class name是和 Java的Type类似的东西，就是namespace。Assembly是Java没有的东西。Assembly Name的属性包括Name, Version, Culture, PublicKey(Token)。 如果两个Assembly有一个属性不一样，那么它们就认为是不一样的。两个Type如果有同样的Class name，但是Assembly Name不一样的话，它们也认为是不一样的。

CLR的Type引用总是包括Class name 和 Assembly Name的。所以CLR在寻找一个Type的时候，主要是寻找Assembly。找到了Assembly之后，CLR看看这个Assembly里有没有这个Class。有的话就万事大吉，没有的话就是TypeLoadException。

CLR区别Type主要靠Assembly。Assembly有PublicKey(Token)。别人无法冒充你。所以象Java里的namespace的问题就不存在了。Assembly有自己的Version。你要第二版的时候CLR不会装载第一版给你。所以象Java Class Loader那样的限制，责任和问题都不存在了。从这个角度上讲，这正是为什么CLR里没有象Java Class Loader那样的东西。

但并不是说你就不能动态装载了，而是说你动态装载的时候就不用考虑那些垃圾了。

关于共享的问题，因为没有了你自己的Class loader，所以任何Type都可以直接对话。同样的Type也不会因为Class Loader不一样而被认为不一样。

CLR Loader有自己的规则怎么寻找Assembly。它先看GAC，再看你的程序目录。都没有的话它还有一个AssemblyResolveEvent去问你，看你能不能提供。如果你要动态装载的话，你有Assembly.Load, Assembly.LoadFrom, Assembly.LoadFile, Assembly.Load(byte[])。你可以提供privateBin，你也可以提供codebase hint。你还能有policy。总之，你想从哪找，就可以去哪找。细节问题我就不多说了。你可以去看MSDN，Suzanne的Blog，Alan的Blog，和我的英文Blog.

结论，CLR Loader远比Java Class Loader要Secure, Powerful and Flexible。

美国科幻影片《时间机器》剧照

　　澳大利亚科学家保罗·戴维斯的科学研究包括黑洞、量子场论、宇宙起源、意识的本质和生命起源等诸多涉及人类和宇宙本源的问题。他曾经带领一个研 究小组，得出光速在变化的结论；他曾经写过不少关于时间的书；如果他设想的一切都成为现实，人可以回到过去见一见自己的老爷爷、老奶奶，也可以到未来看一 看自己的重孙子、重孙女，关于时空的基本定义就要改写，物理学的根基就会动摇。　　

　　“有限时间旅行”肯定可行

　　人类乐于梦想，也盼望知晓过去和未来。若要将历史和命运活生生地展现在我们面前，时间旅行似乎是个最为简便的方法。然而，时间机器的制造仍然停 留在幻想阶段，而即便是幻想，也多产生于小说家和电影特效师的手中，少有人知科学家对时间机器的具体构想。保罗·戴维斯是享誉世界的理论物理学家，他决定 当第一个吃螃蟹的人，尝试制造时间机器，哪怕只是“理论上的制造”。

　　要制造时间机器首先需要弄清楚的问题是，你想回到过去还是飞到未来。在戴维斯看来，在时间中到未来旅行是很容易的，如果你接近于光速运动或者身处强大的引力场中，会感到时间流逝得比其他人更缓慢——你进入了他们的未来。

　　飞向未来的旅行又叫“有限时间旅行”。保罗·戴维斯在《怎样制造时间机器》的开头非常肯定的提出，“有限时间旅行”是可行的，但回到任何时代的“无限时间旅行”只是“有可能可行”。

　　虫洞：回到过去的关键

　　爱因斯坦的相对论允许这一旅行发生在特定的时空结构里：一个旋转的宇宙，一个旋转的柱体，以及非常著名的虫洞——一条贯穿空间和时间的隧道。也就是说，只要能够建造一个稳定的虫洞，就可以跨越时间和空间。那么，到底什么是虫洞？它和黑洞有什么联系呢？

　　在斯蒂芬·霍金的《时间简史》里有这样的解释：虫洞是连接宇宙遥远区域间的时空细管，它可以把平行的宇宙或者婴儿宇宙连接起来，并提供时间旅行的可能性。而黑洞是时空的一个区域，那里引力是如此之强，以至于任何东西，甚至光都不能从该处逃逸出来。

　　在戴维斯的计划里，建造一个虫洞要分3步：

　　第一步，寻找或建立一个虫洞，开辟一个隧道用来连接太空中两个不同的区域。

　　第二步，使虫洞稳定下来。由量子产生的负能量，虫洞便允许信号和物体安全地穿越它。负能量会抵制虫洞变为密度无穷大或接近无穷大。换句话说，它阻止了虫洞演变成黑洞。

　　第三步是牵引虫洞。一艘具有高度先进技术的太空船将虫洞的入口互相分离开。如果两个端口都放置在空间中合适的地方，那么时间差将保持恒定状态。 假设这一差值是10年，一名宇航员从一个方向穿越虫洞，他将跳到10年后的未来，反之，宇航员若是从另一方向穿越虫洞，他将跳到10年前的过去。

　　时间机器的悖论

　　假如技术上的诸多难题都被克服了，时间机器的生产将会打开充满悖论的潘多拉盒子。对于这些，戴维斯也表现出一丝忧虑。他说：“我本人不打算把我 描述的时间旅行和其它控制自然的行为区分开来。所有的技术都在以某种方式干预自然。在一些科幻小说里，有人通过回到过去而改变了现状。这种事出现在小说里 当然无伤大雅，但如果发生在现实中会带来严重的伦理问题。谁给你回到过去改变历史的权利？”

　　既然过去、现在和未来紧密联系，过去能影响现在，那么现在影响过去在逻辑上也说得通。如果不能想干什么就干什么，人好不容易有了在时间中穿梭的自由，却又失去了行动的自由，眼睁睁看着历史从身边滑过，却无力改变什么，岂不是一个巨大的损失？

　　对此，戴维斯给出的解决方案是“多宇宙”理论——世界不是只有一个，而是有许多平行的世界。你回到过去，但那不是你自己的世界，而是和你的历史相似的世界。这样，即便你打死了自己的母亲，她在那个世界也的确死了，但当你回到未来时，她依然活得好好的。

　　这种想法近乎疯狂，但除了戴维斯外还有许多著名物理学家相信有平行世界的存在。在《时间简史》里，霍金这样说：解决时间旅行的其他可能的方法是 选择历史假想。其思想是，当时间旅行者回到过去，他就进入和历史记载不同的另外的一个历史中去。这样，他们可以自由地行动，不受和原先的历史相一致的约 束。如果这样，回到过去和做一场梦又有什么区别？