BlogJava-jojo's blog－－快乐忧伤都与你同在-文章分类-Linux 技术相关

每日一记 2009/07/09 Sed Command Tutorial/Example

Blog of JoJo — Thu, 09 Jul 2009 06:45:00 GMT

SED Tutorial

The sed utility is an "editor"
It is also noninteractive. This means you have to insert commands to be executed on the data at the command line or in a script to be processed.
sed accepts a series of commands and executes them on a file (or set of files).
sed fittingly stands for stream editor.
It can be used to change all occurrences of "SAD" to "SED" or "New York" to "Newport."
The stream editor is ideally suited to performing repetitive edits that would take considerable time if done manually.

How sed Works

The sed utility works by sequentially reading a file, line by line, into memory. It then performs all actions specified for the line and places the line back in memory to dump to the terminal with the requested changes made. After all actions have taken place to this one line, it reads the next line of the file and repeats the process until it is finished with the file. As mentioned, the default output is to display the contents of each line on the screen. Two important factors come into play here—first, the output can be redirected to another file to save the changes; second, the original file, by default, is left unchanged. The default is for sed to read the entire file and make changes to each line within it. It can, however, be restricted to specified lines as needed.

The syntax for the utility is:

sed [options] '{command}' [filename]

In this tutorial we will walk through the most commonly used commands and options and illustrate how they work and where they would be appropriate for use.

The Substitute Command

One of the most common uses of the sed utility, and any similar editor, is to substitute one value for another. To accomplish this, the syntax for the command portion of the operation is:

's/{old value}/{new value}/'

Thus, the following illustrates how "lion" can be changed to "eagle" very simply:

$ echo The lion group will meet on Tuesday after school | sed 



's/lion/eagle/'



The eagle group will meet on Tuesday after school



$

Notice that it is not necessary to specify a filename if input is being derived from the output of a preceding command—the same as is true for awk, sort, and most other LinuxUNIX command-line utility programs.

Multiple Changes

If multiple changes need to be made to the same file or line, there are three methods by which this can be accomplished. The first is to use the "-e" option, which informs the program that more than one editing command is being used. For example:

$ echo The lion group will meet on Tuesday after school | sed -e '



s/lion/eagle/' -e 's/after/before/'



The eagle group will meet on Tuesday before school



$

This is pretty much the long way of going about it, and the "-e" option is not commonly used to any great extent. A more preferable way is to separate command with semicolons:

$ echo The lion group will meet on Tuesday after school | sed '



s/lion/eagle/; s/after/before/'



The eagle group will meet on Tuesday before school 



$

Notice that the semicolon must be the next character following the slash. If a space is between the two, the operation will not successfully complete and an error message will be returned. These two methods are well and good, but there is one more method that many administrators prefer. The key thing to note is that everything between the two apostrophes (' ') is interpreted as sed commands. The shell program reading in the commands will not assume you are finished entering until the second apostrophe is entered. This means that the command can be entered on multiple lines—with Linux changing the prompt from PS1 to a continuation prompt (usually ">")—until the second apostrophe is entered. As soon as it is entered, and Enter pressed, the processing will take place and the same results will be generated, as the following illustrates:

$ echo The lion group will meet on Tuesday after school | sed '



> s/lion/eagle/



> s/after/before/'



The eagle group will meet on Tuesday before school



$

Global Changes

Let's begin with a deceptively simple edit. Suppose the message that is to be changed contains more than one occurrence of the item to be changed. By default, the result can be different than what was expected, as the following illustrates:

$ echo The lion group will meet this Tuesday at the same time



as the meeting last Tuesday | sed 's/Tuesday/Thursday/'



The lion group will meet this Thursday at the same time



as the meeting last Tuesday 



$

Instead of changing every occurrence of "Tuesday" for "Thursday," the sed editor moves on after finding a change and making it, without reading the whole line. The majority of sed commands function like the substitute one, meaning they all work for the first occurrence of the chosen sequence in each line. In order for every occurrence to be substituted, in the event that more than one occurrence appears in the same line, you must specify for the action to take place globally:

$ echo The lion group will meet this Tuesday at the same time



as the meeting last Tuesday | sed 's/Tuesday/Thursday/g'



The lion group will meet this Thursday at the same time



as the meeting last Thursday



$

Bear in mind that this need for globalization is true whether the sequence you are looking for consists of only one character or a phrase.

sed can also be used to change record field delimiters from one to another. For example, the following will change all tabs to spaces:

sed 's/	/ /g'

where the entry between the first set of slashes is a tab, while the entry between the second set is a space. As a general rule, sed can be used to change any printable character to any other printable character. If you want to change unprintable characters to printable ones—for example, a bell to the word "bell"—sed is not the right tool for the job (but tr would be).

Sometimes, you don't want to change every occurrence that appears in a file. At times, you only want to make a change if certain conditions are met—for example, following a match of some other data. To illustrate, consider the following text file:

$ cat sample_one



one     1



two     1



three   1



one     1



two     1



two     1



three   1



$

Suppose that it would be desirable for "1" to be substituted with "2," but only after the word "two" and not throughout every line. This can be accomplished by specifying that a match is to be found before giving the substitute command:

$ sed '/two/ s/1/2/' sample_one



one     1



two     2



three   1



one     1



two     2



two     2



three   1



$

And now, to make it even more accurate:

$ sed '



> /two/ s/1/2/



> /three/ s/1/3/' sample_one



one     1



two     2



three   3



one     1



two     2



two     2



three   3



$

Bear in mind once again that the only thing changed is the display. If you look at the original file, it is the same as it always was. You must save the output to another file to create permanence. It is worth repeating that the fact that changes are not made to the original file is a true blessing in disguise—it lets you experiment with the file without causing any real harm, until you get the right commands working exactly the way you expect and want them to.

The following saves the changed output to a new file:

$ sed '



> /two/ s/1/2/



> /three/ s/1/3/' sample_one > sample_two

The output file has all the changes incorporated in it that would normally appear on the screen. It can now be viewed with head, cat, or any other similar utility.

Script Files

The sed tool allows you to create a script file containing commands that are processed from the file, rather than at the command line, and is referenced via the "-f" option. By creating a script file, you have the ability to run the same operations over and over again, and to specify far more detailed operations than what you would want to try to tackle from the command line each time.

Consider the following script file:

$ cat sedlist



/two/ s/1/2/



/three/ s/1/3/



$

It can now be used on the data file to obtain the same results we saw earlier:

$ sed -f sedlist sample_one



one     1



two     2



three   3



one     1



two     2



two     2



three   3



$

Notice that apostrophes are not used inside the source file, or from the command line when the "-f" option is invoked. Script files, also known as source files, are invaluable for operations that you intend to repeat more than once and for complicated commands where there is a possibility that you may make an error at the command line. It is far easier to edit the source file and change one character than to retype a multiple-line entry at the command line.

Restricting Lines

The default is for the editor to look at, and for editing to take place on, every line that is input to the stream editor. This can be changed by specifying restrictions preceding the command. For example, to substitute "1" with "2" only in the fifth and sixth lines of the sample file's output, the command would be:

$ sed '5,6 s/1/2/' sample_one



one     1



two     1



three   1



one     1



two     2



two     2



three   1



$

In this case, since the lines to changes were specifically specified, the substitute command was not needed. Thus you have the flexibility of choosing which lines to changes (essentially, restricting the changes) based upon matching criteria that can be either line numbers or a matched pattern.

Prohibiting the Display

The default is for sed to display on the screen (or to a file, if so redirected) every line from the original file, whether it is affected by an edit operation or not; the "-n" parameter overrides this action. "-n" overrides all printing and displays no lines whatsoever, whether they were changed by the edit or not. For example:

$ sed -n -f sedlist sample_one



$







$ sed -n -f sedlist sample_one > sample_two



$ cat sample_two



$

In the first example, nothing is displayed on the screen. In the second example, nothing is changed, and thus nothing is written to the new file—it ends up being empty. Doesn't this negate the whole purpose of the edit? Why is this useful? It is useful only because the "-n" option has the ability to be overridden by a print command (-p). To illustrate, suppose the script file were modified to now resemble the following:

$ cat sedlist



/two/ s/1/2/p



/three/ s/1/3/p



$

Then this would be the result of running it:

$ sed -n -f sedlist sample_one



two     2



three   3



two     2



two     2



three   3



$

Lines that stay the same as they were are not displayed at all. Only the lines affected by the edit are displayed. In this manner, it is possible to pull those lines only, make the changes, and place them in a separate file:

$ sed -n -f sedlist sample_one > sample_two



$







$ cat sample_two



two     2



three   3



two     2



two     2



three   3



$

Another method of utilizing this is to print only a set number of lines. For example, to print only lines two through six while making no other editing changes:

$ sed -n '2,6p' sample_one



two     1



three   1



one     1



two     1



two     1



$

All other lines are ignored, and only lines two through six are printed as output. This is something remarkable that you cannot do easily with any other utility. head will print the top of a file, and tail will print the bottom, but sed allows you to pull anything you want to from anywhere.

Deleting Lines

Substituting one value for another is far from the only function that can be performed with a stream editor. There are many more possibilities, and the second-most-used function in my opinion is delete. Delete works in the same manner as substitute, only it removes the specified lines (if you want to remove a word and not a line, don't think of deleting, but think of substituting it for nothing—s/cat//).

The syntax for the command is:

'{what to find} d'

To remove all of the lines containing "two" from the sample_one file:

$ sed '/two/ d' sample_one



one     1



three   1



one     1



three   1



$

To remove the first three lines from the display, regardless of what they are:

$ sed '1,3 d' sample_one



one     1



two     1



two     1



three   1



$

Only the remaining lines are shown, and the first three cease to exist in the display. There are several things to keep in mind with the stream editor as they relate to global expressions in general, and as they apply to deletions in particular:

The up carat (^) signifies the beginning of a line, thus

sed '/^two/ d' sample_one

would only delete the line if "two" were the first three characters of the line.

The dollar sign ($) represents the end of the file, or the end of a line, thus

sed '/two$/ d' sample_one

would delete the line only if "two" were the last three characters of the line.

The result of putting these two together:

sed '/^$/ d' {filename}

deletes all blank lines from a file. For example, the following substitutes "1" for "2" as well as "1" for "3" and removes any trailing lines in the file:

$ sed '/two/ s/1/2/; /three/ s/1/3/; /^$/ d' sample_one



one     1



two     1



three   1



one     1



two     2



two     2



three   1



$

A common use for this is to delete a header. The following command will delete all lines in a file, from the first line through to the first blank line:

sed '1,/^$/ d' {filename}

Appending and Inserting Text

Text can be appended to the end of a file by using sed with the "a" option. This is done in the following manner:

$ sed '$a



> This is where we stop



> the test' sample_one



one     1



two     1



three   1



one     1



two     1



two     1



three   1



This is where we stop



the test



$

Within the command, the dollar sign ($) signifies that the text is to be appended to the end of the file. The backslashes () are necessary to signify that a carriage return is coming. If they are left out, an error will result proclaiming that the command is garbled; anywhere that a carriage return is to be entered, you must use the backslash.

To append the lines into the fourth and fifth positions instead of at the end, the command becomes:

$ sed '3a



> This is where we stop



> the test' sample_one



one     1



two     1



three   1



This is where we stop



the test



one     1



two     1



two     1



three   1



$

This appends the text after the third line. As with almost any editor, you can choose to insert rather than append if you so desire. The difference between the two is that append follows the line specified, and insert starts with the line specified. When using insert instead of append, just replace the "a" with an "i," as shown below:

$ sed '3i



> This is where we stop



> the test' sample_one



one     1



two     1



This is where we stop



the test



three   1



one     1



two     1



two     1



three   1



$

The new text appears in the middle of the output, and processing resumes normally after the specified operation is carried out.

Reading and Writing Files

The ability to redirect the output has already been illustrated, but it needs to be pointed out that files can be read in and written out to simultaneously during operation of the editing commands. For example, to perform the substitution and write the lines between one and three to a file called sample_three:

$ sed '



> /two/ s/1/2/



> /three/ s/1/3/



> 1,3 w sample_three' sample_one



one     1



two     2



three   3



one     1



two     2



two     2



three   3



$







$ cat sample_three



one     1



two     2



three   3



$

Only the lines specified are written to the new file, thanks to the "1,3" specification given to the w (write) command. Regardless of those written, all lines are displayed in the default output.

The Change Command

In addition to substituting entries, it is possible to change the lines from one value to another. The thing to keep in mind is that substitute works on a character-for-character basis, whereas change functions like delete in that it affects the entire line:

$ sed '/two/ c



> We are no longer using two' sample_one



one     1



We are no longer using two



three   1



one     1



We are no longer using two



We are no longer using two



three   1



$

Working much like substitute, the change command is greater in scale—completely replacing the one entry for another, regardless of character content, or context. At the risk of overstating the obvious, when substitute was used, then only the character "1" was replaced with "2," while when using change, the entire original line was modified. In both situations, the match to look for was simply the "two."

Change All but...

With most sed commands, the functions are spelled out as to what changes are to take place. Using the exclamation mark, it is possible to have the changes take place everywhere but those specified—completely reversing the default operation.

For example, to delete all lines that contain the phrase "two," the operation is:

$ sed '/two/ d' sample_one



one     1



three   1



one     1



three   1



$

And to delete all lines except those that contain the phrase "two," the syntax becomes:

$ sed '/two/ !d' sample_one



two     1



two     1



two     1



$

If you have a file that contains a list of items and want to perform an operation on each of the items in the file, then it is important that you first do an intelligent scan of those entries and think about what you are doing. To make matters easier, you can do so by combining sed with any iteration routine (for, while, until).

As an example, assume you have a text file named "animals" with the following entries:

pig
horse
elephant
cow
dog
cat

And you want to run the following routine:

#mcd.ksh



for I in $*



do



echo Old McDonald had a $I



echo E-I, E-I-O



done

The result will be that each line is printed at the end of "Old McDonald has a." While this is correct for the majority of the entries, it is grammatically incorrect for the "elephant" entry, as the result should be "an elephant" rather than "a elephant." Using sed, you can scan the output from your shell file for such grammatical errors and correct them on the fly, by first creating a file of commands:

#sublist



/ a a/ s/ a / an /



/ a e/ s/ a / an /



/a i/ s / a / an /



/a o/ s/ a / an /



/a u/ s/ a / an /

and then executing the process as follows:

$ sh mcd.ksh 'cat animals' | sed -f sublist

Now, after the mcd script has been run, sed will scan the output for anywhere that the single letter a (space, "a," space) is followed by a vowel. If such exists, it will change the sequence to space, "an," space. This corrects the problem before it ever prints on the screen and ensures that editors everywhere sleep easier at night. The result is:

Old McDonald had a pig
E-I, E-I-O
Old McDonald had a horse
E-I, E-I-O
Old McDonald had an elephant
E-I, E-I-O
Old McDonald had a cow
E-I, E-I-O
Old McDonald had a dog
E-I, E-I-O
Old McDonald had a cat
E-I, E-I-O

Quitting Early

The default is for sed to read through an entire file and stop only when the end is reached. You can stop processing early, however, by using the quit command. Only one quit command can be specified, and processing will continue until the condition calling the quit command is satisfied.

For example, to perform substitution only on the first five lines of a file and then quit:

$ sed '



> /two/ s/1/2/



> /three/ s/1/3/



> 5q' sample_one



one     1



two     2



three   3



one     1



two     2



$

The entry preceding the quit command can be a line number, as shown, or a find/matching command like the following:

$ sed '



> /two/ s/1/2/



> /three/ s/1/3/



> /three/q' sample_one



one     1



two     2



three   3



$

You can also use the quit command to view lines beyond a standard number and add functionality that exceeds those in head. For example, the head command allows you to specify how many of the first lines of a file you want to see—the default number is ten, but any number can be used from one to ninety-nine. If you want to see the first 110 lines of a file, you cannot do so with head, but you can with sed:

sed 110q filename

Handling Problems

The main thing to keep in mind when dealing with sed is how it works. It works by reading one line in, performing all the tasks it knows to perform on that one line, and then moving on to the next line. Each line is subjected to every editing command given.

This can be troublesome if the order of your operations is not thoroughly thought out. For example, suppose you need to change all "two" entries to "three" and all "three" to "four":

$ sed '



> /two/ s/two/three/



> /three/ s/three/four/' sample_one



one     1



four     1



four   1



one     1



four     1



four     1



four   1



$

The very first "two" read was changed to "three." It then meets the criteria established for the next edit and becomes "four." The end result is not what was wanted—there are now no entries but "four" where there should be "three" and "four."

When performing such an operation, you must pay diligent attention to the manner in which the operations are specified and arrange them in an order in which one will not clobber another. For example:

$ sed '



> /three/ s/three/four/



> /two/ s/two/three/' sample_one



one     1



three     1



four   1



one     1



three     1



three     1



four   1



$

This works perfectly, since the "three" value is changed prior to "two" becoming "three."

Labels and Comments

Labels can be placed inside sed script files to make it easier to explain what is transpiring, once the files begin to grow in size. There are a variety of commands that relate to these labels, and they include:

: The colon signifies a label name. For example:

         :HERE

Labels beginning with the colon can be addressed by "b" and "t" commands.

b {label} Works as a "goto" statement, sending processing to the label preceded by a colon. For example,

    b HERE

sends processing to the line

    :HERE

If no label is specified following the b, processing goes to the end of the script file.

t {label} Branches to the label only if substitutions have been made since the last input line or execution of a "t" command. As with "b," if a label name is not given, processing moves to the end of the script file.

# The pound sign as the first character of a line causes the entire line to be treated as a comment. Comment lines are different from labels and cannot be branched to with b or t commands.

Blog of JoJo 2009-07-09 14:45 发表评论

每日一记 2009/06/16 Linux的时间设置与同步 (NTP)

Blog of JoJo — Tue, 16 Jun 2009 09:54:00 GMT

Network Time Protocol (NTP) 也是RHCE新增的考试要求. 学习的时候也顺便复习了一下如何设置Linux的时间,现在拿出来和大家分享
设置NTP服务器不难但是NTP本身是一个很复杂的协议. 这里只是简要地介绍一下实践方法
和上次一样,下面的实验都在RHEL5上运行

1. 时间和时区

如果有人问你说现在几点? 你看了看表回答他说晚上8点了. 这样回答看上去没有什么问题,但是如果问你的这个人在欧洲的话那么你的回答就会让他很疑惑,因为他那里还太阳当空呢.

这里就有产生了一个如何定义时间的问题. 因为在地球环绕太阳旋转的24个小时中,世界各地日出日落的时间是不一样的.所以我们才有划分时区(timezone) 的必要,也就是把全球划分成24个不同的时区. 所以我们可以把时间的定义理解为一个时间的值加上所在地的时区(注意这个所在地可以精确到城市)

地理课上我们都学过格林威治时间(GMT), 它也就是0时区时间. 但是我们在计算机中经常看到的是UTC. 它是Coordinated Universal Time的简写. 虽然可以认为UTC和GMT的值相等(误差相当之小),但是UTC已经被认定为是国际标准,所以我们都应该遵守标准只使用UTC

那么假如现在中国当地的时间是晚上8点的话,我们可以有下面两种表示方式

20:00 CST
12:00 UTC

这里的CST是Chinese Standard Time,也就是我们通常所说的北京时间了. 因为中国处在UTC+8时区,依次类推那么也就是12:00 UTC了.

为什么要说这些呢(呵呵这里不是地理论坛吧...)?

第一,不管通过任何渠道我们想要同步系统的时间,通常提供方只会给出UTC+0的时间值而不会提供时区(因为它不知道你在哪里).所以当我们设置系统时间的时候,设置好时区是首先要做的工作
第二,很多国家都有夏令时(我记得小时候中国也实行过一次),那就是在一年当中的某一天时钟拨快一小时(比如从UTC+8一下变成UTC+9了),那么同理到时候还要再拨慢回来.如果我们设置了正确的时区,当需要改变时间的时候系统就会自动替我们调整

现在我们就来看一下如何在Linux下设置时区,也就是time zone

2. 如何设置Linux Time Zone

在Linux下glibc提供了我们事先编译好的许多timezone文件, 他们就放在/usr/share/zoneinfo这个目录下,这里基本涵盖了大部分的国家和城市

代码:

# ls -F /usr/share/zoneinfo/



Africa/      Chile/   Factory    Iceland      Mexico/   posix/      Universal



America/     CST6CDT  GB         Indian/      Mideast/  posixrules  US/



Antarctica/  Cuba     GB-Eire    Iran         MST       PRC         UTC



Arctic/      EET      GMT        iso3166.tab  MST7MDT   PST8PDT     WET



Asia/        Egypt    GMT0       Israel       Navajo    right/      W-SU



Atlantic/    Eire     GMT-0      Jamaica      NZ        ROC         zone.tab



Australia/   EST      GMT+0      Japan        NZ-CHAT   ROK         Zulu



Brazil/      EST5EDT  Greenwich  Kwajalein    Pacific/  Singapore



Canada/      Etc/     Hongkong   Libya        Poland    Turkey



CET          Europe/  HST        MET          Portugal  UCT

在这里面我们就可以找到自己所在城市的time zone文件. 那么如果我们想查看对于每个time zone当前的时间我们可以用zdump命令

代码:

# zdump Hongkong



Hongkong  Fri Jul  6 06:13:57 2007 HKT

那么我们又怎么来告诉系统我们所在time zone是哪个呢? 方法有很多,这里举出两种

第一个就是修改/etc/localtime这个文件,这个文件定义了我么所在的local time zone.
我们可以在/usr/share/zoneinfo下找到我们的time zone文件然后拷贝去到/etc/localtimezone(或者做个symbolic link)

假设我们现在的time zone是BST(也就是英国的夏令时间,UTC+1)

代码:

# date



Thu Jul  5 23:33:40 BST 2007

我们想把time zone换成上海所在的时区就可以这么做

代码:

# ln -sf /usr/share/zoneinfo/posix/Asia/Shanghai /etc/localtime



# date



Fri Jul  6 06:35:52 CST 2007

这样时区就改过来了(注意时间也做了相应的调整)

第二种方法也就设置TZ环境变量的值. 许多程序和命令都会用到这个变量的值. TZ的值可以有多种格式,最简单的设置方法就是使用tzselect命令

代码:

# tzselect



...



TZ='America/Los_Angeles';export TZ

tzselect会让你选择所在的国家和城市(我省略了这些步骤),最后输出相应的TZ变量的值.那么如果你设置了TZ的值之后时区就又会发生变化

代码:

# date



Thu Jul  5 15:48:11 PDT 2007

通过这两个例子我们也可以发现TZ变量的值会override /etc/localtime. 也就是说当TZ变量没有定义的时候系统才使用/etc/localtime来确定time zone. 所以你想永久修改time zone的话那么可以把TZ变量的设置写入/etc/profile里

好了现在我们知道怎么设置时区了,下面我们就来看看如何设置Linux的时间吧

3. Real Time Clock(RTC) and System Clock

说道设置时间这里还要明确另外一个概念就是在一台计算机上我们有两个时钟:一个称之为硬件时间时钟(RTC),还有一个称之为系统时钟(System Clock)

硬件时钟是指嵌在主板上的特殊的电路, 它的存在就是平时我们关机之后还可以计算时间的原因
系统时钟就是操作系统的kernel所用来计算时间的时钟. 它从1970年1月1日00:00:00 UTC时间到目前为止秒数总和的值在Linux下系统时间在开机的时候会和硬件时间同步(synchronization),之后也就各自独立运行了

那么既然两个时钟独自运行,那么时间久了必然就会产生误差了,下面我们来看一个例子

代码:

# date



Fri Jul  6 00:27:13 BST 2007



# hwclock --show



Fri 06 Jul 2007 12:27:17 AM BST  -0.968931 seconds

通过hwclock --show命令我们可以查看机器上的硬件时间(always in local time zone), 我们可以看到它和系统时间还是有一定的误差的, 那么我们就需要把他们同步

如果我们想要把硬件时间设置成系统时间我们可以运行以下命令

代码:

# hwclock --hctosys

反之,我们也可以把系统时间设置成硬件时间

代码:

# hwclock --systohc

那么如果想设置硬件时间我们可以开机的时候在BIOS里设定.也可以用hwclock命令

代码:

# hwclock --set --date="mm/dd/yy hh:mm:ss"

如果想要修改系统时间那么用date命令就最简单了

代码:

# date -s "dd/mm/yyyy hh:mm:ss"

现在我们知道了如何设置系统和硬件的时间. 但问题是如果这两个时间都不准确了怎么办? 那么我们就需要在互联网上找到一个可以提供我们准确时间的服务器然后通过一种协议来同步我们的系统时间,那么这个协议就是NTP了. 注意接下去我们所要说的同步就都是指系统时间和网络服务器之间的同步了

4. 设置NTP Server前的准备

其实这个标题应该改为设置"NTP Relay Server"前的准备更加合适. 因为不论我们的计算机配置多好运行时间久了都会产生误差,所以不足以给互联网上的其他服务器做NTP Server. 真正能够精确地测算时间的还是原子钟. 但由于原子钟十分的昂贵,只有少部分组织拥有, 他们连接到计算机之后就成了一台真正的NTP Server. 而我们所要做的就是连接到这些服务器上同步我们系统的时间,然后把我们自己的服务器做成NTP Relay Server再给互联网或者是局域网内的用户提供同步服务

好了,前面讲了一大堆理论,现在我们来动手实践一下吧. 架设一个NTP Relay Server其实非常简单,我们先把需要的RPM包装上

代码:

# rpm -ivh ntp-4.2.2p1-5.el5.rpm

那么第一步我们就要找到在互联网上给我们提供同步服务的NTP Server

http://www.pool.ntp.org是NTP的官方网站,在这上面我们可以找到离我们城市最近的NTP Server. NTP建议我们为了保障时间的准确性,最少找两个个NTP Server
那么比如在英国的话就可以选择下面两个服务器

0.uk.pool.ntp.org
1.uk.pool.ntp.org

它的一般格式都是number.country.pool.ntp.org

第二步要做的就是在打开NTP服务器之前先和这些服务器做一个同步,使得我们机器的时间尽量接近标准时间. 这里我们可以用ntpdate命令

代码:

# ntpdate 0.uk.pool.ntp.org



6 Jul 01:21:49 ntpdate[4528]: step time server 213.222.193.35 offset -38908.575181 sec



# ntpdate 0.pool.ntp.org



6 Jul 01:21:56 ntpdate[4530]: adjust time server 213.222.193.35 offset -0.000065 sec

假如你的时间差的很离谱的话第一次会看到调整的幅度比较大,所以保险起见可以运行两次. 那么为什么在打开NTP服务之前先要手动运行同步呢?

1. 因为根据NTP的设置,如果你的系统时间比正确时间要快的话那么NTP是不会帮你调整的,所以要么你把时间设置回去,要么先做一个手动同步
2. 当你的时间设置和NTP服务器的时间相差很大的时候,NTP会花上较长一段时间进行调整.所以手动同步可以减少这段时间

5. 配置和运行NTP Server

现在我们就来创建NTP的配置文件了, 它就是/etc/ntp.conf. 我们只需要加入上面的NTP Server和一个driftfile就可以了

代码:

# vi /etc/ntp.conf



server 0.uk.pool.ntp.org



server 1.uk.pool.ntp.org



driftfile /var/lib/ntp/ntp.drift

非常的简单. 接下来我们就启动NTP Server,并且设置其在开机后自动运行

代码:

# /etc/init.d/ntpd/start



# chkconfig --level 35 ntpd on

6. 查看NTP服务的运行状况

现在我们已经启动了NTP的服务,但是我们的系统时间到底和服务器同步了没有呢? 为此NTP提供了一个很好的查看工具: ntpq (NTP query)

我建议大家在打开NTP服务器后就可以运行ntpq命令来监测服务器的运行.这里我们可以使用watch命令来查看一段时间内服务器各项数值的变化

代码:

# watch ntpq -p



Every 2.0s: ntpq -p                                  Sat Jul  7 00:41:45 2007







remote           refid      st t when poll reach   delay   offset  jitter



==============================================================================



+193.60.199.75   193.62.22.98     2 u   52   64  377    8.578   10.203 289.032



*mozart.musicbox 192.5.41.41      2 u   54   64  377   19.301  -60.218 292.411

现在我就来解释一下其中的含义

remote: 它指的就是本地机器所连接的远程NTP服务器

refid: 它指的是给远程服务器(e.g. 193.60.199.75)提供时间同步的服务器

st: 远程服务器的级别. 由于NTP是层型结构,有顶端的服务器,多层的Relay Server再到客户端. 所以服务器从高到低级别可以设定为1-16. 为了减缓负荷和网络堵塞,原则上应该避免直接连接到级别为1的服务器的.

t: 这个.....我也不知道啥意思^_^

when: 我个人把它理解为一个计时器用来告诉我们还有多久本地机器就需要和远程服务器进行一次时间同步

poll: 本地机和远程服务器多少时间进行一次同步(单位为秒). 在一开始运行NTP的时候这个poll值会比较小,那样和服务器同步的频率也就增加了,可以尽快调整到正确的时间范围.之后poll值会逐渐增大,同步的频率也就会相应减小

reach: 这是一个八进制值,用来测试能否和服务器连接.每成功连接一次它的值就会增加

delay: 从本地机发送同步要求到服务器的round trip time

offset: 这是个最关键的值, 它告诉了我们本地机和服务器之间的时间差别. offset越接近于0,我们就和服务器的时间越接近

jitter: 这是一个用来做统计的值. 它统计了在特定个连续的连接数里offset的分布情况. 简单地说这个数值的绝对值越小我们和服务器的时间就越精确

那么大家细心的话就会发现两个问题: 第一我们连接的是0.uk.pool.ntp.org为什么和remote server不一样? 第二那个最前面的+和*都是什么意思呢?

第一个问题不难理解,因为NTP提供给我们的是一个cluster server所以每次连接的得到的服务器都有可能是不一样.同样这也告诉我们了在指定NTP Server的时候应该使用hostname而不是IP

第二个问题和第一个相关,既然有这么多的服务器就是为了在发生问题的时候其他的服务器还可以正常地给我们提供服务.那么如何知道这些服务器的状态呢? 这就是第一个记号会告诉我们的信息

*
它告诉我们远端的服务器已经被确认为我们的主NTP Server,我们系统的时间将由这台机器所提供

+
它将作为辅助的NTP Server和带有*号的服务器一起为我们提供同步服务. 当*号服务器不可用时它就可以接管

-
远程服务器被clustering algorithm认为是不合格的NTP Server

x
远程服务器不可用

了解这些之后我们就可以实时监测我们系统的时间同步状况了

7. NTP安全设置

运行一个NTP Server不需要占用很多的系统资源,所以也不用专门配置独立的服务器,就可以给许多client提供时间同步服务, 但是一些基本的安全设置还是很有必要的
那么这里一个很简单的思路就是第一我们只允许局域网内一部分的用户连接到我们的服务器. 第二个就是这些client不能修改我们服务器上的时间

在/etc/ntp.conf文件中我们可以用restrict关键字来配置上面的要求

首先我们对于默认的client拒绝所有的操作

代码:

restrict default kod nomodify notrap nopeer noquery

然后允许本机地址一切的操作

代码:

restrict 127.0.0.1

最后我们允许局域网内所有client连接到这台服务器同步时间.但是拒绝让他们修改服务器上的时间

代码:

restrict 192.168.1.0 mask 255.255.255.0 nomodify

把这三条加入到/etc/ntp.conf中就完成了我们的简单配置. NTP还可以用key来做authenticaiton,这里就不详细介绍了

8. NTP client的设置

做到这里我们已经有了一台自己的Relay Server.如果我们想让局域网内的其他client都进行时间同步的话那么我们就都应该照样再搭建一台Relay Server,然后把所有的client都指向这两台服务器(注意不要把所有的client都指向Internet上的服务器). 只要在client的ntp.conf加上这你自己的服务器就可以了

代码:

server ntp1.leonard.com



server ntp2.leonard.com

9. 一些补充和拾遗

1. 配置文件中的driftfile是什么?
我们每一个system clock的频率都有小小的误差,这个就是为什么机器运行一段时间后会不精确. NTP会自动来监测我们时钟的误差值并予以调整.但问题是这是一个冗长的过程,所以它会把记录下来的误差先写入driftfile.这样即使你重新开机以后之前的计算结果也就不会丢失了

2. 如何同步硬件时钟?
NTP一般只会同步system clock. 但是如果我们也要同步RTC的话那么只需要把下面的选项打开就可以了

代码:

# vi /etc/sysconfig/ntpd



SYNC_HWCLOCK=yes

10. 参考资料

1. http://www.freebsd.org/cgi/man.cgi?q...conf&sektion=5
不知为什么Redhat没有ntp.conf的man page.费了好大劲才从FreeBSD上找到了.

2. http://www.eecis.udel.edu/~mills/ntp/html/index.html
官方的NTP文档

3. http://tldp.org/HOWTO/TimePrecision-HOWTO/index.html
The Linux Documentation Project上的NTP HOWTO

4. www.pool.ntp.org/
全球NTP服务器提供站

11. 说明

顺便说一下, 大家也许会注意到标准时间的英文是Coordinated Universal Time, 为什么缩写会是UTC呢？

在Wiki上给出的定论是英国人把它叫做Coordinated Universal Time。但是法语中它叫temps universel coordonné, 缩写也就是TUC.两个国家争执不休，最后妥协起见就把它叫做UTC了

其实我看了挺有感触的。英语虽然是通用语言，但是问题在于现在很多核心技术全都掌握在老美手中，所以每有一个新名次都是会以英文来定义, 我觉得过多的英语依赖也是阻碍Linux在中国普及的一个因素. 要是中国能在新的领域有所突破的话也绝对可以起上自己的名字

呵呵，乱说一通。只是想到有一天要是老外问我们北京时间英语怎么说的时候不知道会有多骄傲啊

Blog of JoJo 2009-06-16 17:54 发表评论

每日一记 2009/06/06 linux服务器配置之Sendmail配置

Blog of JoJo — Sat, 06 Jun 2009 11:22:00 GMT

今天我们来配置一下简单的sendmail服务。。。呵呵、、、真的简单的配置。。。

rpm -q sendmail

还需安装这三个包
sendmail-cf-8.12.8-4.i386

sendmail-devel-8.12.8-4.i

sendmail-doc-8.12.8-4.i38

安装好后，我开始配置文件，，，，，

修改/etc/mail/local-hosts-name文件
[root@localhost named]# cat /etc/mail/local-host-names
# local-host-names - include all aliases for your machine here.
xuwini.com

更改/etc/mail/sendmail.mc文件，修改下列地方：
DaemonPortsOptions=Port=smtp,Addr=127.0.0.1, Name=MTA 更改为：
DaemonPortsOptions=Port=smtp,Addr=yourip或者0.0.0.0, Name=MTA
然后m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf

修改 /etc/rc.d/rc.local
加入一行 /usr/sbin/saslauthd -a shadow

用户管理
认证的配置：修改/etc/mail/sendmail.mc中的字段，取消“TRUST_AUTH_MECH”一行和下一行“define”处的注释。然后m4 /etc/ mail/sendmail.mc>/etc/mail/sendmail.cf。
[root@localhost named]# chkconfig --list saslauthd 开启认证
saslauthd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@localhost named]# chkconfig --level 35 saslauthd on

建立用户帐号
[root@localhost named]# groupadd mailuser
[root@localhost named]# adduser -g mailuser -s /bin/bash xuwin
[root@localhost named]# adduser -g mailuser -s /sbin/nologin xxx
[root@localhost named]# passwd xuwin
[root@localhost named]# passwd xxx 密码都是123

【修改/etc/aliases文件实现邮件转发和邮件列表：
admin: xxx 为邮件用户xxx设置别名admin
testgroup: xuwin,xxx 实现群发发给testgroup的邮件发给xuwin 和 xxx 以上2个可以分别测试
#newaliases 】  -------对于我们简单的邮件体系没什么大的用处，个人觉得

访问控制设置
更改/etc/mail/accesss文件，增加：
[root@localhost named]# cat /etc/mail/access
# Check the /usr/share/doc/sendmail/README.cf file for a description
# of the format of this file. (search for access_db in that file)
# The /usr/share/doc/sendmail/README.cf is part of the sendmail-doc
# package.
#
# by default we allow relaying from localhost...
localhost.localdomain RELAY
localhost RELAY
127.0.0.1 RELAY
xuwin.com RELAY
完成后makemap hash /etc/mail/access.db < /etc/mail/access进行数据库更新。

#service sendmail restart

已经成功进入。。。。
我们试着来写一份简单的信。。。
那么我在服务器上收信看看。。。。晕倒。。把密码打出啦。。。
[root@localhost root]# telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 localhost.localdomain ESMTP Sendmail 8.12.8/8.12.8; Sun, 10 Aug 2008 20:47:0
6 +0800
mail from: root@xuwin.com
250 2.1.0 root@xuwin.com... Sender ok
rcpt to: xuwin@xuwin.com
250 2.1.5 xuwin@xuwin.com... Recipient ok
data
354 Enter mail, end with "." on a line by itself
test
hi.baidu.com/proxuwin
xuwin.com
.
250 2.0.0 m7ACl6B7004271 Message accepted for delivery
quit
221 2.0.0 localhost.localdomain closing connection
Connection closed by foreign host.
检查日志[root@localhost root]# tail /var/log/maillog

你如果想在客户机上收发邮件，那么还需跟着我做。。呵呵。。。上面算是成功了，我们看看接下来会不会成功。。。。期待。。

检测imap是否安装rpm -q imap
安装rpm -ivh imap-*.rpm

[root@localhost root]# chkconfig imap on
[root@localhost root]# service xinetd restart
停止 xinetd：  确定  ]
启动 xinetd：  确定  ]
[root@localhost root]# grep imap /etc/services
imap            143/tcp         imap2           # Interim Mail Access Proto v2
imap            143/udp         imap2
imap3           220/tcp                         # Interactive Mail Access
imap3           220/udp                         # Protocol v3
imaps           993/tcp                         # IMAP over SSL
imaps           993/udp                         # IMAP over SSL

有错误。。。那么就仔细检查一下，我们刚刚的配置
看了一下，我把网络服务重启了一下。。不知道行不行。。。呵呵
我们继续。。。。我估计是imap没配置好，因为可以发送的。。。。
你们看。。。没出错。。完全是可行的。。。。这个就是我们刚刚发的。。。。
。。。。功夫不负有心人啊。。找到原因了。。。呵呵。。 /sbin/chkconfig imap on。。。。。。是imap没启动。。。晕倒。。。

收到信了把。。。。

当然我们不可能在一台服务器上那么瞎搞，一般都是两台两个域进行邮件交互。。。方法跟这个是一样的。。大家可以试试。。。呵呵。。。。

教程到此，不好意思，浪费大家那么多的时间在找错误上。。。。

Blog of JoJo 2009-06-06 19:22 发表评论

Rsync命令参数详解

Blog of JoJo — Wed, 03 Jun 2009 08:36:00 GMT

Rsync命令参数详解

在对rsync服务器配置结束以后，下一步就需要在客户端发出rsync命令来实现将服务器端的文件备份到客户端来。rsync是一个功能非常强大的工具，其命令也有很多功能特色选项，我们下面就对它的选项一一进行分析说明。Rsync的命令格式可以为以下六种：

rsync [OPTION]... SRC DEST 　　rsync [OPTION]... SRC [USER@]HOST:DEST 　　rsync [OPTION]... [USER@]HOST:SRC DEST 　　rsync [OPTION]... [USER@]HOST::SRC DEST 　　rsync [OPTION]... SRC [USER@]HOST::DEST 　　rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST]

　　对应于以上六种命令格式，rsync有六种不同的工作模式：

　　1)拷贝本地文件。当SRC和DES路径信息都不包含有单个冒号":"分隔符时就启动这种工作模式。如：rsync -a /data /backup

　　2)使用一个远程shell程序(如rsh、ssh)来实现将本地机器的内容拷贝到远程机器。当DST路径地址包含单个冒号":"分隔符时启动该模式。如：rsync -avz *.c foo:src

　　3)使用一个远程shell程序(如rsh、ssh)来实现将远程机器的内容拷贝到本地机器。当SRC地址路径包含单个冒号":"分隔符时启动该模式。如：rsync -avz foo:src/bar /data

　　4)从远程rsync服务器中拷贝文件到本地机。当SRC路径信息包含"::"分隔符时启动该模式。如：rsync -av root@172.16.78.192::www /databack

　　5)从本地机器拷贝文件到远程rsync服务器中。当DST路径信息包含"::"分隔符时启动该模式。如：rsync -av /databack root@172.16.78.192::www

6)列远程机的文件列表。这类似于rsync传输，不过只要在命令中省略掉本地机信息即可。如：rsync -v rsync://172.16.78.192/www

rsync参数的具体解释如下：

-v, --verbose 详细模式输出
-q, --quiet 精简输出模式
-c, --checksum 打开校验开关，强制对文件传输进行校验
-a, --archive 归档模式，表示以递归方式传输文件，并保持所有文件属性，等于-rlptgoD
-r, --recursive 对子目录以递归模式处理
-R, --relative 使用相对路径信息
-b, --backup 创建备份，也就是对于目的已经存在有同样的文件名时，将老的文件重新命名为~filename。可以使用--suffix选项来指定不同的备份文件前缀。
--backup-dir 将备份文件(如~filename)存放在在目录下。
-suffix=SUFFIX 定义备份文件前缀
-u, --update 仅仅进行更新，也就是跳过所有已经存在于DST，并且文件时间晚于要备份的文件。(不覆盖更新的文件)
-l, --links 保留软链结
-L, --copy-links 想对待常规文件一样处理软链结
--copy-unsafe-links 仅仅拷贝指向SRC路径目录树以外的链结
--safe-links 忽略指向SRC路径目录树以外的链结
-H, --hard-links 保留硬链结     -p, --perms 保持文件权限
-o, --owner 保持文件属主信息     -g, --group 保持文件属组信息
-D, --devices 保持设备文件信息    -t, --times 保持文件时间信息
-S, --sparse 对稀疏文件进行特殊处理以节省DST的空间
-n, --dry-run现实哪些文件将被传输
-W, --whole-file 拷贝文件，不进行增量检测
-x, --one-file-system 不要跨越文件系统边界
-B, --block-size=SIZE 检验算法使用的块尺寸，默认是700字节
-e, --rsh=COMMAND 指定使用rsh、ssh方式进行数据同步
--rsync-path=PATH 指定远程服务器上的rsync命令所在路径信息
-C, --cvs-exclude 使用和CVS一样的方法自动忽略文件，用来排除那些不希望传输的文件
--existing 仅仅更新那些已经存在于DST的文件，而不备份那些新创建的文件
--delete 删除那些DST中SRC没有的文件
--delete-excluded 同样删除接收端那些被该选项指定排除的文件
--delete-after 传输结束以后再删除
--ignore-errors 及时出现IO错误也进行删除
--max-delete=NUM 最多删除NUM个文件
--partial 保留那些因故没有完全传输的文件，以是加快随后的再次传输
--force 强制删除目录，即使不为空
--numeric-ids 不将数字的用户和组ID匹配为用户名和组名
--timeout=TIME IP超时时间，单位为秒
-I, --ignore-times 不跳过那些有同样的时间和长度的文件
--size-only 当决定是否要备份文件时，仅仅察看文件大小而不考虑文件时间
--modify-window=NUM 决定文件是否时间相同时使用的时间戳窗口，默认为0
-T --temp-dir=DIR 在DIR中创建临时文件
--compare-dest=DIR 同样比较DIR中的文件来决定是否需要备份
-P 等同于 --partial
--progress 显示备份过程
-z, --compress 对备份的文件在传输时进行压缩处理
--exclude=PATTERN 指定排除不需要传输的文件模式
--include=PATTERN 指定不排除而需要传输的文件模式
--exclude-from=FILE 排除FILE中指定模式的文件
--include-from=FILE 不排除FILE指定模式匹配的文件
--version 打印版本信息
--address 绑定到特定的地址
--config=FILE 指定其他的配置文件，不使用默认的rsyncd.conf文件
--port=PORT 指定其他的rsync服务端口
--blocking-io 对远程shell使用阻塞IO
-stats 给出某些文件的传输状态
--progress 在传输时现实传输过程
--log-format=formAT 指定日志文件格式
--password-file=FILE 从FILE中得到密码
--bwlimit=KBPS 限制I/O带宽，KBytes per second      -h, --help 显示帮助信息

Blog of JoJo 2009-06-03 16:36 发表评论

每日一记 2009/06/03 screen处理后台任务

Blog of JoJo — Wed, 03 Jun 2009 07:15:00 GMT

大家在起后台服务，或者长时间执行某个脚本的时候。

是不是感觉，交互性很差，

有时，想把结果直接在终端上输出，又怕不小心关闭了终端导致进程退出？（有些sshd服务还会设置连接超时，自动退出终端）

现在用 screen 这个指令可以很好的解决跑后台服务的尴尬。

命令其实超简单的：

直接在终端上输入 screen , 这个时候，服务器端会启一个新的终端，但这个终端，与之前的普通终端不一样，它不隶属于 sshd 进程组，这样，当本地终端关闭后，服务器终端不会被 kill。

当然，优点还不止这么些，在服务器终端里执行任务时，你甚至可以随时地切换到本地终端做些其他事情，然后，要回去时，再恢复到刚才已经打开的服务器终端里，如果刚才的任务没有结束，还可以继续执行任务。

操作步骤：

首先，进入 screen -S sessionname终端。(sessionname是为了分辨你的session)

然后按 ctrl + a ，再按 d 键暂时退出终端。

当要返回时，先查看刚才的终端进程ID， screen -list

或直接

screen -r xx (刚才的sessionname)就可以了

当然，当你开了很多个session后，打算关闭几个session，可以进入到session后，exit 一下就可以了.

总的来说， screen是可以完全替代 nohup 的，并且本身提供了较复杂的功能，但是我认为，刚才那些简单的功能足以应付日常运作。

如果对该指令感兴趣的朋友，推荐一篇文章：

http://www.ibm.com/developerworks/cn/linux/l-cn-screen/

补充：

A: secureCRT链接linux服务器时，网络断线了，但是服务器上的tty还没退出，我想重新连接到原来那个tty，可以做到么？

B: 可以啊，你可以安装一个Screen，就可以享受他给你带来的方便了。

A：如何创建一个虚拟shell环境？

B：screen -S MyScr （其中“MyScr”是你为这个虚拟shell环境起的名字，可以自定义）
输入回车之后，你就可以在这个虚拟的shell环境中工作了，你工作的内容都会被一直保留下来。试试吧，在里面敲几个命令，运行几个程序，和平常没有两样吧。

A: 如何退出工作环境呢？

B：如果想要退出要怎么办呢？只要按下Ctrl+A，然后按d，就可以退出刚刚建立的虚拟shell环境了（名字是MyScr）
若干时间后，你又想继续刚才的工作了，只要再敲：
screen -r MyScr
就可以看到刚刚的界面了。怎么样，还是很简单适用的吧，呵呵。

A: 可以不给他起名么？

B：当然，你不给这个虚拟shell环境命名也是没问题的，如果只有一个虚拟环境的话，也可以这样用
screen（回车）
工作……退出……
screen -r
继续工作
系统还会默认用PID号码表识screen虚拟的shell环境。
例如，我直接用screen命令建立了一个虚拟环境，退出之后，我想查看虚拟环境的情况：

A：如何察看当前有哪些Screen工作环境呢？

B：screen -list
输出应该是类似下面的：
There is a screen on:
25202.pts-1.firewallX (Detached)
1 Socket in /tmp/screens/S-root.
其中：
“25202”是这个虚拟环境的PID。不信的话，可以ps查看一下，呵呵~
“pts-1”是说你的ssh客户端登录的系统端口号是pts-1
“firewallX”是我这台主机的名字
如果你建立了很多虚拟环境，又没有为他们命名的话，就只能用PID来识别他们了。（记住这个PID号码太烦了吧，还是名字好！）
例如，我现在有两个screen建立的虚拟环境，我输入：
screen -list
输出为：
There are screens on:
25202.pts-1.firewallX (Detached)
25403.pts-1.firewallX (Detached)
2 Sockets in /tmp/screens/S-root.
那么，我如果想进入第二个虚拟环境的话，我可以用什么命令呢？答案如下：
screen -r 25403

A：如何创建新的screen呢？

B：如果我登录进去之后，用ctrl-a c：再创建一个新的虚拟Shell环境，那么这个环境就是在PID为25403的虚拟环境里面的子虚拟环境。

A：如何彻底退出一个screen工作环境呢？

B：Ctrl-D

A：如何在进入工作环境的时候就自动的运行screen呢？
B：可以在~/.bash_profiler里最下面一行添加一句screen

注意事项：关闭secureCRT之前，请先使用 Ctrl-D 退出screen

Blog of JoJo 2009-06-03 15:15 发表评论

遍历两个日期之间天数的算法

Blog of JoJo — Mon, 01 Jun 2009 03:10:00 GMT

package pkg.chart;

import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Calendar;
import java.util.Date;

public class Test {
public static void main(String[] args) throws ParseException {
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
Long startM = sdf.parse("2009-1-14").getTime();
Long endM = sdf.parse("2010-1-14").getTime();
long result = (endM - startM) / (24 * 60 * 60 * 1000);
System.out.println("差:" + result + "天");

Date startDate = sdf.parse("2009-01-14");
Calendar startTime = Calendar.getInstance();
startTime.clear();
startTime.setTime(startDate);
for (int i = 0; i < (int)result;i++) {
String str = startTime.get(Calendar.YEAR) + "-"
+ startTime.get(Calendar.MONTH) + "-"
+ startTime.get(Calendar.DAY_OF_MONTH);
System.out.println(str);
startTime.add(Calendar.DAY_OF_YEAR, 1);
}
}
}

package demo;

import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Calendar;
import java.util.Date;
import java.util.GregorianCalendar;

/**
* 遍历两个日期之间天数的算法
*
*/
public class MyTest {
public static void main(String[] args) throws ParseException {
  String start = "2007-01-27";
  String end = "2008-03-04";
  //字符串转换成日期
  SimpleDateFormat format=new SimpleDateFormat("yyyy-MM-dd");
  Date startDate=format.parse(start);
  Calendar startTime=Calendar.getInstance();
  startTime.clear();
  startTime.setTime(startDate);
  int startYear = startTime.get(Calendar.YEAR);
  int startMonth = startTime.get(Calendar.MONTH);
  int startDay = startTime.get(Calendar.DAY_OF_MONTH);
  Date endDate=format.parse(end);
  Calendar endTime=Calendar.getInstance();
  endTime.clear();
  endTime.setTime(endDate);
  int endYear = endTime.get(Calendar.YEAR);
  int endMonth = endTime.get(Calendar.MONTH);
  int endDay = endTime.get(Calendar.DAY_OF_MONTH);
  System.out.println("注意西方的月份从0到11，中国的月份从1到12");
  System.out.println("下面输入的是中国的日期.注意其中的转换问题");
  System.out.println("start date : " + start);
  System.out.println("end date : " + end);

  int count = 0;
  for (int x = startYear; x <= endYear; x++) {
   //罗马历法产生年份公元1582年
   int gregorianCutoverYear = 1582;
   boolean isLeapYear = x >= gregorianCutoverYear ?
     ((x%4 == 0) && ((x%100 != 0) || (x%400 == 0))) :
      (x%4 == 0);
   //判断是否是闰年
   //java方法
   //boolean isLeapYear = (new GregorianCalendar()).isLeapYear(x);

   String isBigYear = "是平年";
   if (isLeapYear) {
    isBigYear = "是闰年";
   }
   System.out.println(x + "年" + isBigYear);

   //获取开始月的最大天数
   //java方法
   //SimpleDateFormat aFormat=new SimpleDateFormat("yyyy-MM-dd");
   //Date date = aFormat.parse(start);
   //Calendar time = Calendar.getInstance();
   //time.clear();
   //time.setTime(date);
   //int max=time.getActualMaximum(Calendar.DAY_OF_MONTH);//本月份的天数
   //System.out.println(max);

   //获取开始月的最大天数；大月是1，3，5，7，8，10，12；小月是4，6，9，11；特殊月是2
   int max = 0;
   if (startMonth == 1) {
    if (isLeapYear) {
     max = 29;
    }
    if (!isLeapYear) {
     max = 28;
    }
   }
   if (startMonth == 3 || startMonth == 5 || startMonth == 8 || startMonth == 10) {
    max = 30;
   }
   if (startMonth == 0 || startMonth == 2 || startMonth == 4 || startMonth == 6 || startMonth == 7 || startMonth == 9 || startMonth == 11) {
    max = 31;
   }

   //循环每个月
   //如果在日期范围内月份循环时自增到了一年的最后一个月就将月份初始化到一月份
   int y = 0;
   //如果是开始日期的第一个年的月数就从开始月数循环
   if (x == startYear) {
    y = startMonth;
   }
   for (; y < 12; y++) {

    //获取当月的最大天数；大月是1，3，5，7，8，10，12；小月是4，6，9，11；特殊月是2
    max = 0;
    if (y == 1) {
     if (isLeapYear) {
      max = 29;
     }
     if (!isLeapYear) {
      max = 28;
     }
    }
    if (y == 3 || y == 5 || y == 8 || y == 10) {
     max = 30;
    }
    if (y == 0 || y == 2 || y == 4 || y == 6 || y == 7 || y == 9 || y == 11) {
     max = 31;
    }

    int ty = y + 1;
    System.out.println(x + "年" + ty + "月");

    //循环每一天
    int z = 1;
    //如果是开始日期的第一个月的天数就从开始天数循环
    if (x == startYear && y == startMonth) {
     z = startDay;
    }
    for (; z <= max; z++) {
     count++;

     System.out.println( x + "年" + ty + "月" + z + "日");

     if (x == endYear && y == endMonth && z == endDay) {
      break;
     }
    }

    //如果已经遍历过了截至日期的最后月份就中止月份的循环
    if (x == endYear && y == endMonth) {
     break;
    }

   }
  }

  System.out.println(start + " 到 " + end + " 的天数差：" + count);

}

}

Blog of JoJo 2009-06-01 11:10 发表评论

Memcache工作原理

Blog of JoJo — Thu, 07 May 2009 05:51:00 GMT

1 Memcache是什么

Memcache是danga.com的一个项目，最早是为 LiveJournal 服务的，目前全世界不少人使用这个缓存项目来构建自己大负载的网站，来分担数据库的压力。

它可以应对任意多个连接，使用非阻塞的网络IO。由于它的工作机制是在内存中开辟一块空间，然后建立一个HashTable，Memcached自管理这些HashTable。

为什么会有Memcache和memcached两种名称？

其实Memcache是这个项目的名称，而memcached是它服务器端的主程序文件名，

Memcache官方网站：http://www.danga.com/memcached，

2 Memcache工作原理

首先 memcached 是以守护程序方式运行于一个或多个服务器中，随时接受客户端的连接操作，客户端可以由各种语言编写，目前已知的客户端 API 包括 Perl/PHP/Python/Ruby/Java/C#/C 等等。客户端在与 memcached 服务建立连接之后，接下来的事情就是存取对象了，每个被存取的对象都有一个唯一的标识符 key，存取操作均通过这个 key 进行，保存到 memcached 中的对象实际上是放置内存中的，并不是保存在 cache 文件中的，这也是为什么 memcached 能够如此高效快速的原因。注意，这些对象并不是持久的，服务停止之后，里边的数据就会丢失。

与许多 cache 工具类似，Memcached 的原理并不复杂。它采用了C/S的模式，在 server 端启动服务进程，在启动时可以指定监听的 ip，自己的端口号，所使用的内存大小等几个关键参数。一旦启动，服务就一直处于可用状态。Memcached 的目前版本是通过C实现，采用了单进程，单线程，异步I/O，基于事件 (event_based) 的服务方式.使用 libevent 作为事件通知实现。多个 Server 可以协同工作，但这些 Server 之间是没有任何通讯联系的，每个 Server 只是对自己的数据进行管理。Client 端通过指定 Server 端的 ip 地址(通过域名应该也可以)。需要缓存的对象或数据是以 key->value 对的形式保存在Server端。key 的值通过 hash 进行转换，根据 hash 值把 value 传递到对应的具体的某个 Server 上。当需要获取对象数据时，也根据 key 进行。首先对 key 进行 hash，通过获得的值可以确定它被保存在了哪台 Server 上，然后再向该 Server 发出请求。Client 端只需要知道保存 hash(key) 的值在哪台服务器上就可以了。

其实说到底，memcache 的工作就是在专门的机器的内存里维护一张巨大的 hash 表，来存储经常被读写的一些数组与文件，从而极大的提高网站的运行效率。

3 如何使用

建立Manager类

Java代码

package com.alisoft.sme.memcached;
import java.util.Date;
import com.danga.MemCached.MemCachedClient;
import com.danga.MemCached.SockIOPool;
public class MemCachedManager {
// 创建全局的唯一实例
protected static MemCachedClient mcc = new MemCachedClient();
protected static MemCachedManager memCachedManager = new MemCachedManager();
// 设置与缓存服务器的连接池
static {
// 服务器列表和其权重
String[] servers = { "127.0.0.1:11211" };
Integer[] weights = { 3 };
// 获取socke连接池的实例对象
SockIOPool pool = SockIOPool.getInstance();
// 设置服务器信息
pool.setServers(servers);
pool.setWeights(weights);
// 设置初始连接数、最小和最大连接数以及最大处理时间
pool.setInitConn(5);
pool.setMinConn(5);
pool.setMaxConn(250);
pool.setMaxIdle(1000 * 60 * 60 * 6);
// 设置主线程的睡眠时间
pool.setMaintSleep(30);
// 设置TCP的参数，连接超时等
pool.setNagle(false);
pool.setSocketTO(3000);
pool.setSocketConnectTO(0);
// 初始化连接池
pool.initialize();
// 压缩设置，超过指定大小（单位为K）的数据都会被压缩
mcc.setCompressEnable(true);
mcc.setCompressThreshold(64 * 1024);
}
/**
* 保护型构造方法，不允许实例化！
*
*/
protected MemCachedManager() {
}
/**
* 获取唯一实例.
*
* @return
*/
public static MemCachedManager getInstance() {
return memCachedManager;
}
/**
* 添加一个指定的值到缓存中.
*
* @param key
* @param value
* @return
*/
public boolean add(String key, Object value) {
return mcc.add(key, value);
}
public boolean add(String key, Object value, Date expiry) {
return mcc.add(key, value, expiry);
}
public boolean replace(String key, Object value) {
return mcc.replace(key, value);
}
public boolean replace(String key, Object value, Date expiry) {
return mcc.replace(key, value, expiry);
}
/**
* 根据指定的关键字获取对象.
*
* @param key
* @return
*/
public Object get(String key) {
return mcc.get(key);
}
public static void main(String[] args) {
MemCachedManager cache = MemCachedManager.getInstance();
cache.add("hello", 234);
System.out.print("get value : " + cache.get("hello"));
}
}

package com.alisoft.sme.memcached;
import java.util.Date;
import com.danga.MemCached.MemCachedClient;
import com.danga.MemCached.SockIOPool;
public class MemCachedManager {
// 创建全局的唯一实例
protected static MemCachedClient mcc = new MemCachedClient();
protected static MemCachedManager memCachedManager = new MemCachedManager();
// 设置与缓存服务器的连接池
static {
// 服务器列表和其权重
String[] servers = { "127.0.0.1:11211" };
Integer[] weights = { 3 };
// 获取socke连接池的实例对象
SockIOPool pool = SockIOPool.getInstance();
// 设置服务器信息
pool.setServers(servers);
pool.setWeights(weights);
// 设置初始连接数、最小和最大连接数以及最大处理时间
pool.setInitConn(5);
pool.setMinConn(5);
pool.setMaxConn(250);
pool.setMaxIdle(1000 * 60 * 60 * 6);
// 设置主线程的睡眠时间
pool.setMaintSleep(30);
// 设置TCP的参数，连接超时等
pool.setNagle(false);
pool.setSocketTO(3000);
pool.setSocketConnectTO(0);
// 初始化连接池
pool.initialize();
// 压缩设置，超过指定大小（单位为K）的数据都会被压缩
mcc.setCompressEnable(true);
mcc.setCompressThreshold(64 * 1024);
}
/**
* 保护型构造方法，不允许实例化！
*
*/
protected MemCachedManager() {
}
/**
* 获取唯一实例.
*
* @return
*/
public static MemCachedManager getInstance() {
return memCachedManager;
}
/**
* 添加一个指定的值到缓存中.
*
* @param key
* @param value
* @return
*/
public boolean add(String key, Object value) {
return mcc.add(key, value);
}
public boolean add(String key, Object value, Date expiry) {
return mcc.add(key, value, expiry);
}
public boolean replace(String key, Object value) {
return mcc.replace(key, value);
}
public boolean replace(String key, Object value, Date expiry) {
return mcc.replace(key, value, expiry);
}
/**
* 根据指定的关键字获取对象.
*
* @param key
* @return
*/
public Object get(String key) {
return mcc.get(key);
}
public static void main(String[] args) {
MemCachedManager cache = MemCachedManager.getInstance();
cache.add("hello", 234);
System.out.print("get value : " + cache.get("hello"));
}
}

建立数据对象

Java代码

package com.alisoft.sme.memcached;
import java.io.Serializable;
public class TBean implements Serializable {
private static final long serialVersionUID = 1945562032261336919L;
private String name;
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
}

package com.alisoft.sme.memcached;
import java.io.Serializable;
public class TBean implements Serializable {
private static final long serialVersionUID = 1945562032261336919L;
private String name;
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
}

Java代码

创建测试用例
package com.alisoft.sme.memcached.test;
import junit.framework.TestCase;
import org.junit.Test;
import com.alisoft.sme.memcached.MemCachedManager;
import com.alisoft.sme.memcached.TBean;
public class TestMemcached extends TestCase {
private static MemCachedManager cache;
@Test
public void testCache() {
TBean tb = new TBean();
tb.setName("E网打进");
cache.add("bean", tb);
TBean tb1 = (TBean) cache.get("bean");
System.out.println("name=" + tb1.getName());
tb1.setName("E网打进_修改的");
tb1 = (TBean) cache.get("bean");
System.out.println("name=" + tb1.getName());
}
@Override
protected void setUp() throws Exception {
super.setUp();
cache = MemCachedManager.getInstance();
}
@Override
protected void tearDown() throws Exception {
super.tearDown();
cache = null;
}
}
测试结果
[INFO] ++++ serializing for key: bean for class: com.alisoft.sme.memcached.TBean
[INFO] ++++ memcache cmd (result code): add bean 8 0 93 (NOT_STORED)
[INFO] ++++ data not stored in cache for key: bean
[INFO] ++++ deserializing class com.alisoft.sme.memcached.TBean
name=E网打进
[INFO] ++++ deserializing class com.alisoft.sme.memcached.TBean
name=E网打进

Blog of JoJo 2009-05-07 13:51 发表评论

Mencache Introduction

Blog of JoJo — Thu, 07 May 2009 05:50:00 GMT

What is `memcached`?

memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.

Danga Interactive developed memcached to enhance the speed of LiveJournal.com, a site which was already doing 20 million+ dynamic page views per day for 1 million users with a bunch of webservers and a bunch of database servers. memcached dropped the database load to almost nothing, yielding faster page load times for users, better resource utilization, and faster access to the databases on a memcache miss.

How it Works

First, you start up the memcached daemon on as many spare machines as you have. The daemon has no configuration file, just a few command line options, only 3 or 4 of which you'll likely use:

# ./memcached -d -m 2048 -l 10.0.0.40 -p 11211

This starts memcached up as a daemon, using 2GB of memory, and listening on IP 10.0.0.40, port 11211. Because a 32-bit process can only address 4GB of virtual memory (usually significantly less, depending on your operating system), if you have a 32-bit server with 4-64GB of memory using PAE you can just run multiple processes on the machine, each using 2 or 3GB of memory.

Porting the Application

Now, in your application, wherever you go to do a database query, first check the memcache. If the memcache returns an undefined object, then go to the database, get what you're looking for, and put it in the memcache:

Perl Example (see APIs page)


sub get_foo_object {

my $foo_id = int(shift);

my $obj = $::MemCache->get("foo:$foo_id");

return $obj if $obj;



$obj = $::db->selectrow_hashref("SELECT .... FROM foo f, bar b ".

"WHERE ... AND f.fooid=$foo_id");

$::MemCache->set("foo:$foo_id", $obj);

return $obj;

}

(If your internal API was already clean enough, you should only have to do this in a few spots. Start with the queries that kill your database the most, then move to doing as much as possible.)

You'll notice the data structure the server provides is just a dictionary. You assign values to keys, and you request values from keys.

Now, what actually happens is that the API hashes your key to a unique server. (You define all the available servers and their weightings when initializing the API) Alternatively, the APIs also let you provide your own hash value. A good hash value for user-related data is the user's ID number. Then, the API maps that hash value onto a server (modulus number of server buckets, one bucket for each server IP/port, but some can be weighted heigher if they have more memory available).

If a host goes down, the API re-maps that dead host's requests onto the servers that are available.

Shouldn't the database do this?

Regardless of what database you use (MS-SQL, Oracle, Postgres, MySQL-InnoDB, etc..), there's a lot of overhead in implementing ACID properties in a RDBMS, especially when disks are involved, which means queries are going to block. For databases that aren't ACID-compliant (like MySQL-MyISAM), that overhead doesn't exist, but reading threads block on the writing threads.

memcached never blocks. See the "Is memcached fast?" question below.

What about shared memory?

The first thing people generally do is cache things within their web processes. But this means your cache is duplicated multiple times, once for each mod_perl/PHP/etc thread. This is a waste of memory and you'll get low cache hit rates. If you're using a multi-threaded language or a shared memory API (IPC::Shareable, etc), you can have a global cache for all threads, but it's per-machine. It doesn't scale to multiple machines. Once you have 20 webservers, those 20 independent caches start to look just as silly as when you had 20 threads with their own caches on a single box. (plus, shared memory is typically laden with limitations)

The memcached server and clients work together to implement one global cache across as many machines as you have. In fact, it's recommended you run both web nodes (which are typically memory-lite and CPU-hungry) and memcached processes (which are memory-hungry and CPU-lite) on the same machines. This way you'll save network ports.

What about MySQL 4.x query caching?

MySQL query caching is less than ideal, for a number of reasons:

MySQL's query cache destroys the entire cache for a given table whenever that table is changed. On a high-traffic site with updates happening many times per second, this makes the the cache practically worthless. In fact, it's often harmful to have it on, since there's a overhead to maintain the cache.
On 32-bit architectures, the entire server (including the query cache) is limited to a 4 GB virtual address space. memcached lets you run as many processes as you want, so you have no limit on memory cache size.
MySQL has a query cache, not an object cache. If your objects require extra expensive construction after the data retrieval step, MySQL's query cache can't help you there.

If the data you need to cache is small and you do infrequent updates, MySQL's query caching should work for you. If not, use memcached.

What about database replication?

You can spread your reads with replication, and that helps a lot, but you can't spread writes (they have to process on all machines) and they'll eventually consume all your resources. You'll find yourself adding replicated slaves at an ever-increasing rate to make up for the diminishing returns each additional slave provides.

The next logical step is to horizontally partition your dataset onto different master/slave clusters so you can spread your writes, and then teach your application to connect to the correct cluster depending on the data it needs.

While this strategy works, and is recommended, more databases (each with a bunch of disks) statistically leads to more frequent hardware failures, which are annoying.

With memcached you can reduce your database reads to a mere fraction, leaving the databases to mainly do infrequent writes, and end up getting much more bang for your buck, since your databases won't be blocking themselves doing ACID bookkeeping or waiting on writing threads.

Is `memcached` fast?

Very fast. It uses libevent to scale to any number of open connections (using epoll on Linux, if available at runtime), uses non-blocking network I/O, refcounts internal objects (so objects can be in multiple states to multiple clients), and uses its own slab allocator and hash table so virtual memory never gets externally fragmented and allocations are guaranteed O(1).

What about race conditions?

You might wonder: "What if the get_foo() function adds a stale version of the Foo object to the cache right as/after the user updates their Foo object via update_foo()?"

While the server and API only have one way to get data from the cache, there exists 3 ways to put data in:

set -- unconditionally sets a given key with a given value (update_foo() should use this)
add -- adds to the cache, only if it doesn't already exist (get_foo() should use this)
replace -- sets in the cache only if the key already exists (not as useful, only for completeness)

Additionally, all three support an expiration time.

Blog of JoJo 2009-05-07 13:50 发表评论

how to monitor ibm mq from nagios

Blog of JoJo — Wed, 06 May 2009 07:51:00 GMT

This was one of the search terms that found an article here… I hadn’t addressed this directly, but I use Nagios to monitor my company’s server environment, and specifically implemented that monitoring for IBM Websphere MQ.

For MQ, I run nagios monitoring against queue depth and processes. I installed three plugins to run against WebSphere. Of these one was developed for my company’s needs (qdepth), one was changed slightly (channels) and the last debugged, found not to actually measure accurately, and not resolved (message age).

Here’s the nagios console for the websphere MQ server. “message age” in the second qdepth check service title is deceptive - actually checking qdepth…

This is the commands section from the nrpe.cfg file on the WebSphere MQ server.

command[check_mq_channel]=/usr/local/nagios/libexec/check_mq_channel.sh $ARG1$ $ARG2$ command[check_mq_msgage]=/usr/local/nagios/libexec/check_mq_msgage.sh $ARG1$ $ARG2$ $ARG3$ $ARG4$ command[wmq_check_qdepth]=/usr/local/nagios/libexec/wmq_check_qdepth.pl $ARG1$ $ARG2$ $ARG3$

Of these we only really using qdepth monitoring. The channels come up triggered, so an inactive state is fine, and the plugin as written only tests for “running”. The message age plugin, as I mentioned, doesn’t actually work.

When I first looked at setting this messaging up and then monitoring it, I searched for “nagios monitoring MQ webshere” and found several pre-written plugins. I took each plugin and tested it for usability and for accurate results and for meeting what we needed for monitoring.

The message age plugin, in testing, actually returned a hard-coded result rather than actually testing and returning a valid answer. I started to fix it, set it aside and haven’t resolved it. I don’t recall the source for the plugin. Check each piece of code you download from the internet - it may have gone through extensive development and testing, or it could just as easily have been hacked together in an hour. Your mileage may seriously vary and I would highly recommend you verify any of this before you bet your job on it.

Here’s the qdepth plugin - I think I wrote or re-wrote this pretty much from scratch, but the original concept for parsing runmcsc came from one of the plugins I downloaded, written by Kyle O’Donnell - the channel plugin has his original author credit intact. This plugin has alerted once to an increasing qdepth, which turned out to be an issue with an SSL certificate.



#! /bin/perl

## wmq_check_qdepth.pl # # nrpe (nagios) script to check websphere qdepth

# uses runmqsc binary # # display queue ('APP.REQUEST') # 8 : display queue ('APP.REQUEST') # AMQ8409: Display Queue details. # QUEUE(APP.REQUEST) TYPE(QLOCAL) # ACCTQ(QMGR) ALTDATE(2008-01-22) # ALTTIME(14.18.23) BOQNAME( ) # BOTHRESH(0) CLUSNL( ) # CLUSTER( ) CLWLPRTY(0) # CLWLRANK(0) CLWLUSEQ(QMGR) # CRDATE(2008-01-22) CRTIME(14.18.23) # CURDEPTH(0) DEFBIND(OPEN) # DEFPRTY(0) DEFPSIST(NO) # DEFSOPT(SHARED) DEFTYPE(PREDEFINED) # DESCR( ) DISTL(NO) # GET(ENABLED) HARDENBO # INITQ( ) IPPROCS(0) # MAXDEPTH(5000) MAXMSGL(4194304) # MONQ(QMGR) MSGDLVSQ(PRIORITY) # NOTRIGGER NPMCLASS(NORMAL) # OPPROCS(0) PROCESS( ) # PUT(ENABLED) QDEPTHHI(80) # QDEPTHLO(20) QDPHIEV(DISABLED) # QDPLOEV(DISABLED) QDPMAXEV(ENABLED) # QSVCIEV(NONE) QSVCINT(999999999) # RETINTVL(999999999) SCOPE(QMGR) # SHARE STATQ(QMGR) # TRIGDATA( ) TRIGDPTH(1) # TRIGMPRI(0) TRIGTYPE(FIRST) # USAGE(NORMAL)

### Variables ###

# test values set if this flag is true (1) ### THIS MUST BE SET TO 0 IN PRODUCTION!!! ### my $test = 0;

# debug flag (adds messages) my $debug = 0; my $LOG = "/tmp/wmq_check_qdepth.pl.log";

# runmqsc binary my $MQSC = "/opt/mqm/bin/runmqsc";

### ARGS ###

# first argument is warn level my $WARN = shift; # second arg is crtitical level my $CRIT = shift;

# third arg is queue name
my $QUEUE = shift;

# set for dev purposes if ($test) { $WARN = 5; $CRIT = 10; $QUEUE = "1A33.EVG.REQUEST"; }

# validate # WARN and CRIT must be greater than 0 and CRIT must be greater than WARN unless (($WARN > 0) && ($CRIT > 0)) { print ("Command Failed: WARN and CRIT levels must be greater than 0"n"); exit 3; } unless ($CRIT > $WARN) { print ("Command Failed: CRIT must be greater than WARN"n"); exit 4; }

### Subs ###

### MAIN ###

# run query my $result = `echo "display queue ('${QUEUE}')" | $MQSC | grep CURDEPTH`; print ("result: $result"n") if $debug; # parse result my @lines = split (""n", $result); # divide into an array by end of line... # each element of the array will contain a single line # set variables my ($PARAM, $VALUE);

for my $line (@lines) { # each line is one or two elements like "QDPLOEV(DISABLED) QDPMAXEV(ENABLED)" # divide those... my ($first, $discard) = split (' ', $line); print (""$first: $first "$discard $discard"n") if $debug; ($PARAM, $VALUE) = split ('"(', $first); $VALUE =~ s/")//; print (""$PARAM: $PARAM "$VALUE: $VALUE"n") if $debug; }

# testing value $VALUE = 13 if $test; # check for $WARN and $CRIT levels, exit 0 as OK, 1 as warn or 2 as critical if ($VALUE == 0) { print ("OK: found qdepth for $QUEUE at 0"n"); exit 0; } elsif ($VALUE < $WARN) { print ("OK: found qdepth for $QUEUE at $VALUE"n"); exit 0; } elsif (($VALUE >= $WARN) && ($VALUE < $CRIT)) { print ("WARN: qdepth of $QUEUE is at $VALUE: exceeds WARN thresh of $WARN"n"); exit 1; } elsif ($VALUE >= $CRIT) { print (”CRITICAL: qdepth for $QUEUE at $VALUE: exceeds CRITICAL thresh of $CRIT"n”); exit 2; }

This is the channel status plugin - I may have re-written the original data gathering runmssc string, but the majority of the plugin remained intact…



#!/bin/ksh

#

# check queue manager status

#

# Kyle O'Donnell 

#

#$Id: check_mq_channel,v 1.2 2007/04/04 14:36:02 kodonnel Exp $

#

# debug

DATE=`date`

LOG=”/tmp/nrpe_check_mq_channel.sh.log”

echo “” >> $LOG

echo $DATE >> $LOG

echo “” >> $LOG

[ $# -ne 2 ] && echo “usage: $0  ” &&  exit 3

channel=$1

qmgr=$2

echo “channel: $channel  qmanager: $qmgr” >> $LOG

RUNMQSC=”/opt/mqm/bin/runmqsc”

chanstatus=`echo “dis chs(${channel}) status” | ${RUNMQSC} ${qmgr} | grep -i “status(running)”`

echo “channel status result:  $chanstatus” >> $LOG

if echo $chanstatus |grep -i “status(running)” > /dev/null 2>&1; then

STATE=0

printf “${channel} on ${qmgr} running”

echo “”

echo “”

else

STATE=2

printf “${channel} on ${qmgr} not running”

echo “”

echo “”

fi

echo “state:  $STATE” >> $LOG

exit $STATE;

Here’s the server.cfg file for the Websphere MQ machine on the nagios server:

define service { use generic-service host_name mq1 service_description Host Alive check_period 24x7 contact_groups unix-administrators notification_period 24x7 check_command check-host-alive }

define service { use generic-service host_name mq1 service_description Sonic Bridge java process check_period 24x7 contact_groups esb-administrators notification_period 24x7 check_command check_unix_proc!mqm!1!java }

define service { use generic-service host_name mq1 service_description SSB queue depth EVGPQM01.DEAD.QUEUE message age check_period 24x7 contact_groups systems-services,help_desk notification_period 24x7 check_command wmq_check_qdepth!1!3!QMGR01!QMGR01.DEAD.QUEUE }

define service { use generic-service host_name mq1 service_description server queue depth APPLICATION.RESPONSE check_period 24x7 contact_groups systems-services,help_desk notification_period 24x7 check_command wmq_check_qdepth!5!10!APPLICATION.RESPONSE }

define service { use generic-service host_name mq1 service_description server queue depth OPPOSITE-QMGR check_period 24x7 contact_groups systems-services,help_desk notification_period 24x7 check_command wmq_check_qdepth!5!10!OPPOSITE-QMGR }

define service { use generic-service host_name mq1 service_description WMQ command server check_period 24x7 contact_groups systems-services,help_desk notification_period 24x7 check_command check_unix_proc!mqm!1!amqpcsea }

define service { use generic-service host_name mq1 service_description WMQ Critical process manager check_period 24x7 contact_groups systems-services,help_desk notification_period 24x7 check_command check_unix_proc!mqm!1!amqzmuc0 }

The strategy is to monitor qdepth and processes specific to IBM WebSphere MQ on the Websphere MQ server, along with the normal UNIX processes and disk space.

— dsm

Share and Enjoy: These icons link to social bookmarking sites where readers can share and discover new web pages.

One Response to “how to monitor ibm mq from nagios”

Nice article! I hadn’t been paying attention and didn’t know there were modules for Nagios to check WMQ.

How about this for checking queue depth…

# Isolate the CURDEPTH element to a line, then strip the attribute so Perl gets only the value my $curdepth = `echo "display queue ('${QUEUE}')" | $MQSC $QMGRNAME | tr ')' '"n' | tr ' ' '"n' | grep CURDEPTH | tr '(' '"n' | grep -v CURDEPTH`;

if ($curdepth == '0') { # Look for literal '0' } elsif ($curdepth > 0) { # Check against $WARN and $CRIT } else { # runmqsc command failed. QMgr down? }

Note that I added the QMgr name to the runmqsc command to handle cases where the QMgr is not set as the default or there is more than one. I also added logic to catch the case where WMQ is down.

In the case of process monitoring, amqzmuc0 is the log formatter and I don’t think it runs in all cases. A better choice might be amqzxma0 which is the execution controller.

– T.Rob

Blog of JoJo 2009-05-06 15:51 发表评论

用nagios来监控网络服务器和网络服务

Blog of JoJo — Wed, 06 May 2009 07:36:00 GMT

摘要: 用nagios来监控网络服务器和网络服务 nagios可以对服务器进行全面的监控，包括服务（apache、mysql、ntp、 dns、disk、qmail和sshd等等）的状态，服务器的状态（up、down等等）。它是一个完全GPL协议的开源软件包，包含有nagios主程序和它的各个插件，配置非常灵活，可以监视的项目很多，可以自定义shell脚本进行监控服务，非常适合大型网络。 g9... 阅读全文

Blog of JoJo 2009-05-06 15:36 发表评论

linux下开机自动启动Oracle脚本

Blog of JoJo — Fri, 24 Apr 2009 09:51:00 GMT

linux下开机自动启动Oracle脚本

最近自己的NC项目跑在Linux环境下。把安装过程慢慢的写下来。

===========linux下开机自动启动Oracle脚本============

#注意：例子中的oralce命令在/home/oracle/oracle/product/10.2.0/db_1/bin目录。

#你可以自己修改成自己的目录。

#!/bin/bash

#alang 2008-10-19

#root

#chkconfig: 345 51 49

#description: starts the oracle dabase deamons

#

#ORA_HOME=/home/oracle/oracle/product/10.2.0/db_1

#ORA_OWNER=oracle

case "$1" in

'start')

echo -n "Starting oracle10g ... "

su - oracle -c "/home/oracle/oracle/product/10.2.0/db_1/bin/dbstart"

echo "Done."

echo "Starting Oracle Listeners ... "

su - oracle -c "/home/oracle/oracle/product/10.2.0/db_1/bin/lsnrctl start"

echo "Done."

#启动web管理界面:http://host_ip_address:1158/em

#su - oracle -c "/home/oracle/oracle/product/10.2.0/db_1/bin/emctl start dbconsole"

touch /var/lock/subsys/oracle10g

echo ""

;;

'stop')

echo -n "shutting down oracle10g ... "

su - oracle -c "/home/oracle/oracle/product/10.2.0/db_1/bin/dbshut"

echo -n "dbshut ok !"

su - oracle -c "/home/oracle/oracle/product/10.2.0/db_1/bin/lsnrctl stop"

echo -n "lsnrctl stop ok !"

#su - oracle -c "/home/oracle/oracle/product/10.2.0/db_1/bin/emctl stop dbconsole"

su - root -c "/home/oracle/ufsoft/stop.sh"

rm -f /var/lock/subsys/oracle10g

echo

;;

'restart')

echo -n "restarting oracle10g ... "

$0 stop

$0 start

echo

;;

*)

echo "Usage: oracle {start|stop|restart} "

exit 1

esac

exit 0

============建立一个名为oralce10g的脚本==========

[root@cwserver ~]# gedit /etc/rc.d/init.d/oracle10g

#复制粘贴本文开头的脚本。或者直接使用附件中的oracle10g文件

#给予执行权限

[root@cwserver ~]# chmod 755 /etc/rc.d/init.d/oracle10g

#注意：例子中的oralce命令在/home/oracle/oracle/product/10.2.0/db_1/bin目录。

你可以自己修改成自己的目录。

============添加到启动、关闭、重启动服务序列中===

$ ln -s /etc/rc.d/init.d/oracle10g /etc/rc.d/rc2.d/S99oracle10g

$ ln -s /etc/rc.d/init.d/oracle10g /etc/rc.d/rc3.d/S99oracle10g

$ ln -s /etc/rc.d/init.d/oracle10g /etc/rc.d/rc5.d/S99oracle10g

$ ln -s /etc/rc.d/init.d/oracle10g /etc/rc.d/rc0.d/K01oracle10g

$ ln -s /etc/rc.d/init.d/oracle10g /etc/rc.d/rc6.d/K01oracle10g

============添加并启动察看服务=============================
$/sbin/chkconfig --add /etc/rc.d/init.d/oracle10g
$/sbin/chkconfig --list /etc/rc.d/init.d/oracle10g

============修改Oracle系统配置文件/etc/oratab=====

$ gedit /etc/oratab

最后一行更改为：

orcl:/home/oracle/oracle/product/10.2.0/db_1:Y

============end==大功告成====================

Blog of JoJo 2009-04-24 17:51 发表评论

Preventing access to .svn folders or a file in Apache

Blog of JoJo — Mon, 20 Apr 2009 07:07:00 GMT

Order Deny,Allow
Deny from All
Satisfy All

RedirectMatch 404 /".svn(/|$)

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
This is straight from the mailing list with grateful thanks to Ryan Schmidt.

Sometimes I can be a bit slow so I hadn't even considered blocking access to the .svn folders on this and other sites I maintain. I have now! There are two solutions:





Order allow,deny



Deny from all











Order allow,deny



Deny from all

and:

RedirectMatch 404 /".svn(/|$)

====================================================================================================

Apache Tips & Tricks: Deny access to some folders

Applies: apache 1.3.x / apache 2.0.x
Required apache module: mod_access
Scope: global server configuration, virtual host, directory, .htaccess
Type: security

Description: How to deny access to certain folders and the files inside them.
Useful: to deny access to certain folders containing private information (log files, source code, password files, etc.). The example shown here will address the question posted by Saul Howard on how to deny access to all the subversion directories (.svn).

I a previous tip (Deny access to certain file types) I have showed how we can deny access to files using a particular filename or all the files with a particular extension or any regexp we can match the files. In this post we will block access to folders, so instead of using the directive we will be using the section.

Allow/Deny Directive in

Let’s see how we can deny access to all the .svn folders that exist on the server.
In order to achieve this we will add the following configuration lines in the appropriate context (either global config, or vhost/directory, or from .htaccess):

Order allow,deny Deny from all
Similar to this we can deny access to other folders we might needâ€¦

Note: this will show a Forbidden page (code 403) even if the folder does not exist and it is just called from the browser in the url.
Another way how this can be quickly accomplished is by using a Rewrite rule:

RewriteRule ^(.*/)?"".svn/ - [F,L]

or using a redirect:

RedirectMatch 404 /"".svn(/|$)

(in this last example I am using 404 as the returned code so this looks like the folder doesn’t exist on the server; of course if you prefer you can return 403 - forbidden code).

Go to:
Main page of all my Apache Tips & Tricks

Blog of JoJo 2009-04-20 15:07 发表评论

Pair programming and agile

Blog of JoJo — Fri, 17 Apr 2009 08:45:00 GMT

Let's change the email subject so Tim can stop scramming on us. :)

Interesting point that Bill pointed out is the availability of the resource. I would like to have a discussion on it. As the whole idea of pair programming is to raise the overall team productivity , so the correct statement should be as follow:

"Solo programming is bad so i don't want too. And we also need to think about the availability of the resource as the productivity of solo programming is really low."

The reason that people always think pair programming is a waste, because:
You are putting two developer to do a task that can be done by one developer.
     1. However, this is totally an illustration , you will realize the truth situation when you dig deeper:
Solo , there are often situation that you are fooled by a simple typo , and spending few days to find it. You may finally solve it yourself, or your bored colleagues come to have a chat with you, look at your screen, and scrum out "You got a typo"...........   The later case is an example of power in pair programming ,i.e., navigating and discover what you overlooked.
     1. Solo, you may fall in sleep, low/no productivity. Pair , your partner will not let's you.
     2. Solo, you play with msn and email . Pair, two developers won't get the chance.
     3. Solo, it's easily that you are moved from development to research, not doing the real thing. Pair, someone will warn you and drag you back.
     4. Solo, you can't think of a solution, not coz you are not good, but you are just not there. Pair, two brains always have a more complete solution.
     5. Solo, you learn something new yourself, wasting 4 days to pick up someone can teach you in an hour. Pair, both got trained and improved all the time.
     6. Of course, it's also correct that we should not drag everyone to pair programming at start for various reason, starting with a small groups of interested developers is much better.

F.Y.I. , I got 11 developers signed up about tonight event.

And please click a confirm on facebook if you can.
http://www.facebook.com/event.php?eid=8836123613Let's change the email subject so Tim can stop scramming on us. :)

Interesting point that Bill pointed out is the availability of the resource. I would like to have a discussion on it. As the whole idea of pair programming is to raise the overall team productivity , so the correct statement should be as follow:

"Solo programming is bad so i don't want too. And we also need to think about the availability of the resource as the productivity of solo programming is really low."

The reason that people always think pair programming is a waste, because:
You are putting two developer to do a task that can be done by one developer.
     1. However, this is totally an illustration , you will realize the truth situation when you dig deeper:
Solo , there are often situation that you are fooled by a simple typo , and spending few days to find it. You may finally solve it yourself, or your bored colleagues come to have a chat with you, look at your screen, and scrum out "You got a typo"...........   The later case is an example of power in pair programming ,i.e., navigating and discover what you overlooked.
     1. Solo, you may fall in sleep, low/no productivity. Pair , your partner will not let's you.
     2. Solo, you play with msn and email . Pair, two developers won't get the chance.
     3. Solo, it's easily that you are moved from development to research, not doing the real thing. Pair, someone will warn you and drag you back.
     4. Solo, you can't think of a solution, not coz you are not good, but you are just not there. Pair, two brains always have a more complete solution.
     5. Solo, you learn something new yourself, wasting 4 days to pick up someone can teach you in an hour. Pair, both got trained and improved all the time.
     6. Of course, it's also correct that we should not drag everyone to pair programming at start for various reason, starting with a small groups of interested developers is much better.

F.Y.I. , I got 11 developers signed up about tonight event.

And please click a confirm on facebook if you can.
http://www.facebook.com/event.php?eid=8836123613

Blog of JoJo 2009-04-17 16:45 发表评论

User Story Estimation Techniques

Blog of JoJo — Fri, 17 Apr 2009 08:43:00 GMT

One of the great things about working as a consultant is the ability to try out many different ideas and adapting your personal favorite process to include things that work. This article gives the details about user story estimation techniques that I've found effective.

Powers of two
Originally I estimated stories as one, two, three, four or as small, medium, large, extra-large. It was always meant to be understood that a medium was twice the size of a small and a large was twice the size of a medium (and so on), but that never seemed to translate well when it came to planning. Then someone recommended to me that I try powers of two. Suddenly we were speaking a language that the business could understand. They knew that an 8 was significantly bigger than a 1. I believe the sizes one, two, four, eight are also much more appropriate. As stories get larger they almost always contain more unknown and risk. A powers of two scale emphasizes the risk associated with large stories.

Use four values
I was once on a project that started with 1, 2, 4, 8 as their estimation values. After the first two estimation sessions less than 5% of the stories were ones and about 30% of the stories were twos. The project manager decided to get rid of the one value because it made his life easier. An interesting thing happened at each subsequent estimation meeting, suddenly only 5% of the stories were twos and many more stories had become fours. I don't think that the developers consciously changed their scale, but developers are conditioned to be skeptical. Few developers are willing to say with certainty that any given story will be as easy as the scale allows. After witnessing this type of behavior on a few different projects, I prefer a minimum of four point values. I also prefer a maximum of 4 point values. After all, it's nothing more than an estimate. If you try to give too much precision to an estimate you'll end up having to account for why you missed the mark. The idea is to get a rough idea, not a rigid plan to live off of.

No averages or numbers not on the scale
Four values allow you to get a rough estimate without spending unnecessary time focusing on precision. Sometimes a story feels larger than a two but smaller than a four. The story should not be estimated as a three. There's really no reason to use a three. The story carries enough risk or unknowns that it is not a two; therefore, it's very likely that it will actually be a four. Using an average or an off scale number can briefly (and unnecessarily) confuse a team member or stakeholder. Also, in the big picture of the project, the occasional uncommon estimate isn't likely to make much of a difference. Keep it simple, stick to the scale.

Vote independently
It's human nature to be influenced by other people. If a technical leader says a story is a two, it's likely that the rest of the team will follow his lead. For this reason I prefer an estimation process that lets each team member vote independently. This can be done by using sheets of paper that no one reveals until everyone is ready. Another option (that I prefer) is to give your estimation rock-paper-scissors (RPS) style. In our estimation meetings we talk about a story until we are ready to estimate then we all "throw" our estimations the same as you would "throw" rock, paper, or scissors. What I mean by "throw our estimation" is that if we think it's a 1 we point 1 finger. Likewise a 2 is two fingers and 4 is four fingers. If you need to throw an 8, you can use both hands.

Take the largest estimate
Even when reminded, developers seem to have a hard time estimating with a team in mind. If a developer thinks they can do the story in 1 day, they throw 1 finger. Unfortunately, that developer may not be available to do the story, and then some other team member is stuck working on a story that they thought was a 2 or even a 4. I prefer to always take the largest estimate thrown by any team member. You may consider this to be sandbagging, but in reality it's likely that each team member has identified different risks and the team member with the largest estimate has probably correctly identified that there is more risk than the other members have thought of.

Taking the largest estimate has additional benefits. If you must agree on a lower estimate then the team member with the larger estimate will need to discuss why they chose a larger value. This discussion can be uncomfortable for developers who are less senior on the team. They may not know how to do something as quickly, based on limited experience with the language or tools. Their concerns are often justified by their skill level, and it would be unfortunate if they felt uncomfortable giving their true estimate because they were afraid to discuss why it was higher.

Any discussion around taking a higher or lower value may lead to the entire team raising their value, or it may lead to that developer uncomfortably lowering their estimate. Either way, you'll need to spend more time talking and you wont have gained--in the end it's consistency that matters. You always know how many stories you expect to get done in an iteration by tracking velocity*. Therefore, even if your estimates are "bloated" so will your velocity be, thus it has no effect on planning.

Finally, taking the largest estimate can help save time in an estimation meeting. If any member of the team believes the story is an 8 he can speak up at any time while discussing the story and announce that he is going to throw an eight. Unless someone else believes that there is a large estimation gap among team members, there's no reason to continue talking about the story since it will ultimately become an eight anyway.

Large estimate gaps
When estimating it's usually the exception that the entire team agrees on the size of a story. Like I previously said, I like to handle the mismatch by always taking the larger estimate. However, sometimes a large gap represents a misunderstanding. For this reason any time there is a two value gap in estimation, additional conversation always occurs (e.g. if a team member throws a 1 and another throws a 4, some clarification needs to occur). Discussing large gaps also ensures that taking the largest estimate has less chance of being abused.

Insufficient information
On occasion a story may need leave the meeting unestimated. It's better to ask for more information than to give an estimate that you are uncomfortable with. An estimate of 8 implies that it's a large story, but you expect it to take twice as long as a 4. Therefore, don't simply estimate ill-defined stories as eights, because you will likely be expected to get it done in the same amount of time as it takes to get two stories estimated as fours completed. The goal of an estimation meeting isn't to estimate all the stories, it's to provide estimates on the stories that provide sufficient information.

Required involvement
No one enjoys estimation meetings (okay, no one I know). In my past projects the fastest reader would read the story aloud, the developers would ask the domain experts questions, and then they would estimate. When the developers weren't asking the domain experts questions, the domain experts usually did other things on their laptops. At first glance I thought this was a good use of their time, but things got missed. Later I joined a project where the manager insisted that we go around the room and make everyone read a story when it was their turn. Suddenly the domain experts were engaged because they were worried about looking silly when it was their turn to read. The meetings became much more valuable due to everyone's involvement.

Pigs and Chickens
In a ham-and-eggs restaurant, the pig is committed but the chicken is simply involved.

I often hear that the business shouldn't influence developer estimates because developers are pigs and the business is full of chickens. I actually think this is a bad analogy. It's more likely that a bad product will get the business team fired than the technology team. I'm sure the business feels just as committed as the developers. However, it is a conflict of interest to let the business interfere with estimates.

It's as simple as this, the business wants to know what functionality they can get in the next iteration. To know what to expect they need estimates. Since the business will not be writing the code, they cannot contribute proper estimates. The more they are involved (in the actual estimation), the less likely it is that they will receive realistic estimates. The best domain experts answer questions in meetings, but never assert in any way the level of effort it will take to complete any given story.

Estimation group size
Teams come in many different sizes. On smaller teams (6 or less) I suggest the entire team attend the estimation session. The many points of view are likely to solidify vision and positively contribute to an estimate. However, I believe there is a point of diminishing returns. Not everyone on a large team needs to be part of estimating each story. Additionally, it's an estimate, 6 people should be just as accurate as 15 people would be. If your team is larger than 6 people I suggest breaking into smaller groups for estimation. In general I like to get at least 3 people to estimate any given story, but no more than 6.

New stories
New stories come in two forms: new feature requests and stories that split. I generally wait to estimate new stories based on their priority. If a story needs to be done in the next iteration, it generally requires an immediate estimate. However, if the new stories aren't going to be played for several iterations it can make sense to hold off until you have enough stories to justify an estimation meeting. I find estimates from estimation meetings to be more reliable, since they come from an environment where everyone is focused solely on estimation. Stories resulting from a split provide an additional complication: they likely already have an estimate. I strongly suggest that the new stories be estimated without taking into consideration the previous estimate. If a story carried enough risk or uncertainty that it required splitting, it's not likely that the estimate is realistic--ignore the original estimate.

No laptops
At least no laptops for developers. Print the story list for everyone, or project the list on the screen, but don't ask the developers to read the story list from their laptops. Laptops almost always find ways to distract developers, thus taking away from the goal of the meeting: Getting valuable estimates.

Required participation
This suggestion is a very important one. In theory, no developer from outside the team should be attending an estimation session. That means that every developer that attends an estimation session will potentially be tasked with working on a story that's being estimated. If a developer is not comfortable estimating a story, then I'm not comfortable with them working on the story. Of course, there are exceptions. I generally give new team members one week to come up to speed before I ask that they participate in an estimation session. But, in general, a developer who refuses to participate in estimation should signal that there's a bigger issue that needs to be resolved.

Stale estimations
Teams change, projects change, and random events occur. Whatever the reason, estimations can get stale. Stale estimations don't help anyone. The development team feels pressure to deliver to stale estimates and the business expects stories to be completed according to projected velocity. It doesn't matter why estimates get stale, what matters is that the estimates are no longer realistic and the plan is no longer reliable. I've never been part of a project where the estimates didn't go stale within 12-24 weeks. It's better to admit that an estimation is stale than it is to plan with inaccurate information. For this reason, I suggest revisiting any estimation that was given more than 12 weeks ago. The estimation will hopefully still be good, but giving the developers an opportunity to speak up given new information is nothing but helpful to the business.

Bribes
This is the easiest suggestion of all: Bring high quality snacks to all estimation meetings. Sugar has been scientifically linked to happiness, and happiness leads to collaboration. It's the simplest and cheapest possible way to make an estimation meeting something to look forward to. Keep in mind though, high quality is the key. If you bring the same snacks that are already sitting in the team room, it's not very exciting. On my last project I went to the bakery and got fresh baked assorted cookies every time I remembered there was a meeting.

Credit
I'd like to give special thanks to Brent Cryder, Dennis Byrne, Fred George, Joe Zenevitch, Mike Ward, and Sean Doran for helping me solidify and evolve these ideas. Just like every other list of people, mine surely leaves out other contributors, please forgive me for leaving you off.

About the author
Jay Fields is a software developer and consultant at ThoughtWorks. He has a passion for discovering and maturing innovative solutions. His most recent work has been in the Domain Specific Language space where he has delivered applications that empowered domain experts to author domain logic. He is also very interested in maturing software design through developer testing and software testing in general.

*Velocity: The number of points you've completed over the life of the project divided by the number of iterations. E.g. If you've completed 20 points over 5 iterations, your velocity is 4. With a velocity of 4 you can expect to get as many stories done in an iteration where their estimates total 4 (e.g. 1 story estimated as a 4, 2 stories estimated as twos, 4 stories estimated as ones, etc)

http://www.infoq.com/articles/agile-estimation-techniques

Blog of JoJo 2009-04-17 16:43 发表评论

Increase file descriptor limit under Linux to prevent java.net.SocketException: Too many open files Increase file descriptor limit under Linux to prevent java.net.SocketException: Too many open files Increase file descriptor limit under Linux to prev

Blog of JoJo — Tue, 14 Apr 2009 02:53:00 GMT

Increase file descriptor limit under Linux to prevent

java.net.SocketException: Too many open files

Get current limit:

ulimit -n

cat /proc/sys/fs/file-nr

The default limit is 1024.

Get current number of open file descriptors:

lsof [-p pid] | wc -l

Increase the limit:

Edit /etc/security/limits.conf:

username hard nofile 32768

Blog of JoJo 2009-04-14 10:53 发表评论

crontab

Blog of JoJo — Tue, 10 Mar 2009 07:42:00 GMT

Once inside the editor, you will want to refer to the fields above in order to schedule a cron job for the appropriate time. Here are some examples:

*/5 * * * * /home/adam/script.sh will execute script.sh every 5 minutes. This will set crontab every 5 minutes.
59 23 * * 1-5 /home/adam/script.sh will execute script.sh every day, monday through friday, at 11:59 p.m.
0 0 * * 0 /home/adam/script.sh will execute script.sh once a week. You could also specify @weekly instead of 0 0 * * 0.

0 23 1 * * /home/adam/script.sh will execute script.sh once a month, on the first, at 11:00 PM. You could also specify @monthly in place of 0 23 1 * *.

Blog of JoJo 2009-03-10 15:42 发表评论

svn中的branch和tag

Blog of JoJo — Tue, 03 Mar 2009 06:57:00 GMT

在SVN中Branch/tag在一个功能选项中，在使用中也往往产生混淆。

在实现上，branch和tag，对于svn都是使用copy实现的，所以他们在默认的权限上和一般的目录没有区别。至于何时用tag，何时用branch，完全由人主观的根据规范和需要来选择，而不是强制的（比如cvs）。

一般情况下，
tag，是用来做一个milestone的，不管是不是release，都是一个可用的版本。这里，应该是只读的。更多的是一个显示用的，给人一个可读（readable）的标记。
branch，是用来做并行开发的，这里的并行是指和trunk进行比较。

比如，3.0开发完成，这个时候要做一个tag，tag_release_3_0，然后基于这个tag做release，比如安装程序等。trunk进入 3.1的开发，但是3.0发现了bug，那么就需要基于tag_release_3_0做一个branch，branch_bugfix_3_0，基于这个branch进行bugfix，等到bugfix结束，做一个tag，tag_release_3_0_1，然后，根据需要决定 branch_bugfix_3_0是否并入trunk。

对于svn还要注意的一点，就是它是全局版本号，其实这个就是一个tag的标记，所以我们经常可以看到，什么什么release，基于xxx项目的2xxxx版本。就是这个意思了。但是，它还明确的给出一个tag的概念，就是因为这个更加的可读，毕竟记住tag_release_1_0要比记住一个很大的版本号容易的多。

branches：分枝

当多个人合作，可能有这样的情况出现：John突然有个想法，跟原先的设计不太一致，可能是功能的添加或者日志格式的改进等等，总而言之，这个想法可能需要花一段时间来完成，而这个过程中，John的一些操作可能会影响Sally的工作，John从现有的状态单独出一个project的话，又不能及时得到Sally对已有代码做的修正，而且独立出来的话，John的尝试成功时，跟原来的合并也存在困难。这时最好的实践方法是使用branches。 John建立一个自己的branch，然后在里面实验，必要的时候从Sally的trunk里取得更新，或者将自己的阶段成果汇集到trunk中。

（svn copy SourceURL/trunk DestinationURL/branchName -m "Creating a private branch of xxxx/trunk." ）

trunk：主干

主干，一般来说就是开发的主要呆的地方，

tag:

在经过了一段时间的开发后，项目到达了一个里程碑阶段，你可能想记录这一阶段的代码的状态，那么你就需要给代码打上标签。

(svn cp file:///svnroot/mojavescripts/trunk file:///svnroot/mojavescripts/tags/mirrorutils_rel_0_0_1
-m "taged mirrorutils_rel_0_0_1")

另有一说，无所谓谁对谁错。

trunk：表示开发时版本存放的目录，即在开发阶段的代码都提交到该目录上。

branches：表示发布的版本存放的目录，即项目上线时发布的稳定版本存放在该目录中。

tags：表示标签存放的目录。

在这需要说明下分三个目录的原因，如果项目分为一期、二期、三期等，那么一期上线时的稳定版本就应该在一期完成时将代码copy到branches上，这样二期开发的代码就对一期的代码没有影响，如新增的模块就不会部署到生产环境上。而branches上的稳定的版本就是发布到生产环境上的代码，如果用户使用的过程中发现有bug，则只要在branches上修改该bug，修改完bug后再编译branches上最新的代码发布到生产环境即可。tags的作用是将在branches上修改的bug的代码合并到trunk上时创建个版本标识，以后branches上修改的bug代码再合并到trunk上时就从tags的version到branches最新的version合并到trunk，以保证前期修改的bug代码不会再合并。

轉自：

http://hi.baidu.com/stevenchan/blog/item/bb70c1951b576d4dd1135e3a.html

http://hi.baidu.com/cc0cc/blog/item/f68cb0581e86add99d82043d.html

Blog of JoJo 2009-03-03 14:57 发表评论

linux 开机自动启动apache 脚本

Blog of JoJo — Mon, 02 Mar 2009 09:30:00 GMT

实例介绍：
   1、在linux下安装了apache 服务（通过下载二进制文件经济编译安装、而非rpm包）、apache 服务启动命令：            /server/apache/bin/apachectl start    。让apache服务运行在运行级别3下面。命令如下：

   1）touch /etc/rc.d/init.d/apache
      vi /etc/rc.d/init.d/apache
      chown -R root /etc/rc.d/init.d/apache
      chmod 700 /etc/rc.d/init.d/apache
      ln -s /etc/rc.d/init.d/apache /etc/rc.d/rc3.d/S60apache   #S 是start的简写、代表启动、K是kill的简写、代表关闭。60数字        代表启动的顺序。（对于iptv系统而言、许多服务都是建立在数据库启动的前提下才能够正常启动的、可以通过该数字就行调整脚本的       启动顺序））

      apache的内容：
      #!/bin/bash
      #Start httpd service
      /usr/local/site/apache/bin/apachectl start

      至此 apache服务就可以在运行级别3下随机自动启动了。（可以结合chkconfig 对启动服务进行相应的调整）

Blog of JoJo 2009-03-02 17:30 发表评论

RRDTool

Blog of JoJo — Sat, 28 Feb 2009 05:12:00 GMT

關於RRDtool
安裝前準備工作
安裝RRDtool
開始使用RRDtool
建立RRD檔
抓取資料
更新RRD檔資料
繪製圖表
其他RRDtool的指令
簡單實作範例
RRDtool備份

關於RRDtool

什麼是RRDtool?相信大部份的人都沒有聽過這個東東....不過，如果問起知不知道有MRTG這套程式，相信就有許多人聽過了。　打個比方來說明︰如果MRTG是一輛車子，那麼RRDtool就是製造車子的工廠了! 簡單講，RRDtool是一套可以把數據畫成圖表的程式，以時間為x軸、流量為y軸，而且可以動態更新圖表的程式，聽起來似乎非常的強大？沒錯，正是因為RRDtool的功能強大以致於它有不太容易學習的缺點，況且坊間有提到RRDtool的書籍可以說是少之又少(目前還沒看過)；網路上找？總共的中文教學文章只有酷學園Abelyang的那一篇。反觀MRTG簡單又功能強大，學習文件頗多，能滿足大部份人的需求，但是如果您對於MRTG的統計方式不能認同，亦或是您有強烈customize統計圖表的需求，RRDtool將會是您最好的選擇！

(本圖取自RRDtool官方網站)

安裝前準備工作

本文以下的敘述皆實作於FreeBSD 6.0系統下，不過如果您的系統不是FreeBSD，不用擔心，大同小異啦 ! 除了一些指令有小小的不同以外...... 此外，因為RRDtool能做出來的只是圖表而已，一般來說，都是以放在WEB的方式讓管理者或是使用者能夠瀏覽，所以，請先安裝好WWW server (如:Apache) 。

安裝rrdtool

在FreeBSD下︰

LEE# cd /usr/ports/net/rrdtool/ //切換到ports中RRDtool的目錄
LEE# make install clean //安裝並清除安裝暫存的檔案

在Debian下:

LEE# apt-get update //更新檔案清單
LEE# apt-get install rrdtool //使用apt安裝RRDtool套件

其他linux則可以到www.rrdtool.org下載Source Code或是Binary檔安裝！

如果安裝過程沒有差錯，RRDtool就安裝完成了! 如何確定RRDtool能正常運作呢? 在console下輸入「rrdtool」，如果有看到以下畫面代表您的RRDtool已經可以正常運作囉！

LEE# rrdtool
RRDtool 1.0.49 Copyright 1997-2004 by Tobias Oetiker

Usage: rrdtool [options] command command_options

Valid commands: create, update, graph, dump, restore,
last, info, fetch, tune, resize, xport

RRDtool is distributed under the Terms of the GNU General
Public License Version 2. (www.gnu.org/copyleft/gpl.html)

For more information read the RRD manpages

LEE#

如果沒有看到，請在確定您的安裝步驟有沒有錯誤！

開始使用RRDtool

先說明一下RRDtool運作的大概流程︰

步驟一︰建立RRD檔，這個檔說來說去就是RRDtool的「專屬」資料庫啦！RRDtool以自有的格式存放流量資料，下面會有比較詳細的說明。

步驟二︰「抓取資料」個人覺得是整個RRDtool最困難的一部分，因為RRDtool的資料是要靠自己弄出來，不若MRTG內建抓資料功能，但是卻因為如此，可以「餵」給RRDtool畫圖的資料彈性也比較大，例如︰snmp查詢結果、系統狀態、網頁中特定數字統計..等等。

步驟三︰抓下來的資料就用「rrdtool update」的指令更新步驟更新的RRD檔的內容，讓圖表能畫出最新的流量。

步驟四︰這就是重點啦！透過「rrdtool graph」的指令來依據RRD檔的資料繪圖，這也是使用者唯一看的到的東西，若規劃的不好會影響使用者閱讀上的困難！

迴圈︰由於要達成動態繪圖的圖表，第二步驟到第四步驟必須不斷的重複執行以維持資料的更新，目前知道要達成迴圈的方法有兩種︰1、在Script中使用迴圈；2、使用cron這個排班程式做排班。

以下將詳細的說明這四個步驟︰

建立RRD檔

RRDtool 建檔語法
rrdtool create filename
[--start|-b start time] // "-s" 代表此RRD檔可以開始紀錄的時間，注意︰要把時間單位轉換成秒，1970/01/01算第一秒，用`date`指令可由系統幫您計算。
[--step|-s step] //意指每筆資料的間隔時間，一般使用者設定'300'，即每五分鐘為一間隔。
[DS:ds-name:DST:heartbeat:min:max]
[RRA:CF:xff:steps:rows]

重要參數說明︰

DS 全名"Data Source"，就是資料來源。這有點像是在RRD檔這個資料庫建一個可以儲存資料的欄位。

例︰DS:telnet:COUNTER:600:0:100000000

DS表示式總共有六個欄位︰第一個欄位宣告這列式為DS表示式；第二個欄位宣告這個DS在RRD檔裡面的"欄位名稱"(Data Source Name)，此例宣告此欄叫做"telnet"(名稱可以自訂)；第三個欄位叫做DST(Data Source Type)，習慣上常用 GAUGE(個別值,像CPU loading) 及COUNTER (累計值,像流量資料) 在產生圖檔時， GAUGE 是 100 就畫100在 Y 軸上；但如果是 COUNTER ，此次值為100，而前一值是 98，則會在Y軸上畫 2；第四個欄位稱做有效期(heartbeat)，範例裡的值為'600'，假設要取12:00的資料，而前後300秒裡的值(11:55-12:05)經過平均或是取最大或最小都算是12:00的有效值；第五個欄位和第六個欄位為這個欄位允許可以存放的最大最小值，此例允許最小為0，最大為100000000，如果不想設限制可以再第五個欄位和第六個欄位以 "U:U"表示。

RRA全名為Round Robin Archive，簡單來說就是其實就是什麼類資料要存幾筆，資料的儲存分成四類，分別為AVERAGE, MIN, MAX, LAST 意即平均值，最大值，最小值，最後一筆。

例︰0.5:1:603

因為我們將 step 定為 300 秒是指若原計算時間點為 12:00 的話，記錄時要以 11:57:30~12:02:30 的平均值為主，這個值若在此時間點內只有一筆資料的話，其意即是平均值，所以此一值即表原 telnet…等共要記錄幾筆，603 是指要存 603 筆，若超過603筆，則最早之一筆將被移出。
0.5:6:603 第三個欄位表示取 6 筆資料(每筆為 step 值，在此意即５分鐘)為平均值( 30 分鐘), 存 603 筆
0.5:24:603 24 即二小時
0.5:288:800 288 即一天

以下為建檔的完整範例︰

$LEE# rrdtool create lee.rrd -s 300 \       //此列為建立'lee.rrd'這個資料檔，step值為300s
?DS:input:COUNTER:600:0:100000000 \       //Data Source第一欄
?DS:output:COUNTER:600:0:100000000 \     //Data Source第-二欄
?RRA:AVERAGE:0.5:1:603 \                         //每隔五分鐘(1*5)存一次資料的平均值
?RRA:AVERAGE:0.5:6:603 \                         //每隔三十分鐘(6*5)存一次資料的平均值
?RRA:AVERAGE:0.5:24:603 \                       //每隔兩小時(24*5)存一次資料的平均值
?RRA:AVERAGE:0.5:288:603 \                     //每隔一天(288*5)存一次資料的平均值
?RRA:MAX:0.5:1:603 \                                  //每隔五分鐘存一次資料的最大值
?RRA:MAX:0.5:6:603 \                                  //每隔三十分鐘存一次資料的最大值
?RRA:MAX:0.5:24:603 \                                //每隔兩小時存一次資料的最大值
?RRA:MAX:0.5:288:603                                //每隔一天存一次資料的最大值
$LEE#

抓取資料

同上面所說，對於不會Shell Script或者是Perl..等可以處理字串的方法的人，這部份算是比較難的。最好先花點時間先稍微閱讀一下此類資訊會比較好上手，以下提供一份不錯的Shell Script的教學文件http://www.study-area.org/cyril/scripts/scripts/。

以下為部分簡單的範例︰

$LEE# snmpget -c public 192.168.0.1 ifInOctets.4
interfaces.ifTable.ifEntry.ifInOctets.4 = Counter32: 13526287

如果某台機器有安裝snmp的服務，則可用snmpget查詢此主機上面的資料，至於可以查詢到什麼資料端看那台機器所提供的MIB功能強不強大。上面的例子使用snmpget的指令嘗試抓取 192.168.0.1 這台主機的位址，"-c"參數代表此台主機的通訊名稱，一般的主機預設為"public"，但是也可以自訂，要了解某台主機的通訊名稱請向管理者詢問。"ifInOctets.4"中的ifInOctets為查詢此台主機的輸入流量，後面的".4"則代表第四個網路介面。

下面的:interfaces.ifTable.ifEntry.ifInOctets.4 = Counter32: 13526287 為主機端回應的結果，因為我們要的資料為「13526287」(此值就是輸入流量)這個值，這時候就是字串處理函式發揮強大功能的時候啦！

$LEE# snmpget -c public 192.168.0.1 ifInOctets.4 | sed -e 's/.*ter32: $.*$/\1/'
13526287

回傳值為「13526287」，是的，這就是我們要的結果。到底是什麼讓前面的那串文字不見而取的我們要的數值呢? 奧妙就在於後面的那串古怪的語法，「sed -e 's/.*ter32: $.*$/\1/'」，不瞭解的話請自行翻閱Shell Script的教學文件，不然用法千百種講也講不完。重要的是這只是我們往後的Script檔裡面的一個小小的範例，下面的範例將會附上一個完整的Script檔，您就能瞭解怎麼把這先抓回來的資料使用在RRDtool上了。

當然，您不一定要使用snmpget，也可以使用snmpwalk、tcpdump..等等抓資料回來分析，說誇張點，凡是有數字會「動」的東西都可以經過處理變成我們要的資料，然後畫成圖表。

更新RRD檔資料

RRD的Update指令大概算是RRDtool裡面最簡單的指令了吧！

語法︰rrdtool update filename [--template|-t ds-name[:ds-name]...] N|timestamp:value[:value...]
這個很好理解,基本上就是根據 DS來更新資料,如上述之lee.rrd,若有需要更新時

$LEE# rrdtool update lee.rrd 1061811856:1199:0:821073 // 後三個欄位分別代表(時間:欄位一值:欄位二值)

上面的 1061811856 即時間值,如果就是要現在的時間值,則可以 N 代表,但要轉換成秒值,通常我們都會以`date +%s`來轉換。可以試試看，在FreeBSD輸入`date +%s`:

LEE# date +%s
1105334125 //從1970/1/1 到現在的秒數

注︰請注意在Linux下和FreeBSD下指令"date"的用法並不相同。如在Linux下"date -d"為轉換某日期成為秒數，而BSD下"date -d"則為系統設定時間，使用不當容易造成系統時間錯誤。

繪製圖表

RRDtool繪圖語法︰(以下僅列出常用的)
rrdtool graph image-filename
-s 繪圖資料的起始時間,預設是一天前,單位是"秒數"
-e 繪圖資料的結束時間,預設是現在,亦可使用 date 方式來達到前三天至昨天圖檔
--no-minor 不要副格線
-t 圖檔標題
-v Y 軸說明
-w 資料區的寬度,資料區指的是數據顯示的部份,而非說明或圖例
-h 資料區的高度
-u Y 軸正值高度
-l Y 軸負值高度
-M 自動調整畫圖的Y軸最大值

DEF 其語法為 DEF:vname:rrd_filename:DS_name:[AVERAGE|MAX..]
這個有點像是上面的DS(Data Source)，亦即在圖表中宣告一個資料來源，必須先宣告，讓以下的圖從這邊取資料。
CDEF 一個虛擬的變數,就是把DEF的資料拿來做加工，其值為 DEF 的某些運算,其運算式需寫成後序，中間以逗號隔開。
EX: a=1+3 寫成 a=1,3 +
http=(input+output)/1024 寫成 http=input,output,+,1024,/

LINE函數其語法︰LINE{1|2|3}:vname[#rrggbb[:legend]]，用這條函式就是您想把資料用線條的方式表示，其中依照線條的粗細又可分為LINE1、LINE2、LINE3三種SIZE；vname這欄填的就是您為DEF這個地方宣告的名稱，這個名稱可自己隨意取；#rrggbb這裡填的就是你想要線條顯示出來的顏色，使用RGB代碼；legend裡面填的是AVERAGE、MAX等表示取值的方式。下面為例子︰

(此圖取自RRDtool完全攻略文章中)

AREA函數其語法︰AREA:vname[#rrggbb[:legend]]，用這條函式就是您想把資料用填充塊狀方塊的方式表示，其他後面的參數請參照LINE的用法，做出來的範例如下︰

(此圖取自RRDtool完全攻略文章中)

STACK函數其語法︰STACK:vname[#rrggbb[:legend]]，則是畫出資料數值至其上的數值,也就是要有資料數值在 STACK 數值之上，用法同上兩個。
如果使用 AREA/STACK 時需特別注意圖蓋圖的問題,一定要先畫大的值,再畫小的值,才會有層次的效果,不然,最大的數據若最後畫,會直接把小的資料蓋過去，導致小值無法顯示出來。

COMMENT 說明欄字,如上圖的的"Last Updated'字樣為使用COMMENT:"Last Updated"指令產生，可以用 \n 等換行符號。
GPRINT:GPRINT GPRINT:vname:CF:format vname 即DEF 中的 vname,而 CF 看你要輸出的文字是AVERAGE/MAX/MIN/LAST 等數值,format 如同 printf 中的格式,
做出來如下圖例︰

其他少用的RRDtool的指令

rrdtool info rrdfile.rrd - 顯示出rrd檔目前的資料儲存方式(如每幾分鐘球平均或算最大值....等等)。

rrdtool dump rrdfile.rrd > filename.xml - 把rrd二進位檔的資料以XML的資料格式匯出。

rrdtool restore filename.xml filename.rrd [--range-check|-r] - 把rrd匯出的XML檔匯入rrd的二進位檔內。

rrdtool fetch rrdfile.rrd CF [--resolution|-r resolution] [--start|-s start] [--end|-e end] - 抓取rrd資料檔內的特定資料。

以上指令詳細使用方式請參照官方網站︰www.rrdtool.org

簡單實作範例

寫了那麼多，還是來個簡單的從頭到尾的範例吧！

步驟一︰先在console下建立RRD檔，

$LEE# rrdtool create test.rrd -s 300 DS:input:COUNTER:600:0:100000000 \ //檔名請自訂，可以把test.rrd改掉
?DS:output:COUNTER:600:0:100000000 \
?RRA:AVERAGE:0.5:1:603 \
?RRA:AVERAGE:0.5:6:603 \
?RRA:AVERAGE:0.5:24:603 \
?RRA:AVERAGE:0.5:288:603 \
?RRA:MAX:0.5:1:603 \
?RRA:MAX:0.5:6:603 \
?RRA:MAX:0.5:24:603 \
?RRA:MAX:0.5:288:603
$LEE# ls
test.rrd //這樣就是建立成功了

步驟二︰把下面的Script碼複製，用文書編輯器新建一個檔，把以下內容放到那個檔(假設叫做 test.sh)，並且依照黃色字的指示在相關變數裡面填入您自己主機的相關資料，填完後把黃色的註解刪除(不然執行時會出現錯誤)，最後打 "sh test.sh"(看您的檔取什麼名稱)執行即可。

#!/bin/sh //宣告使用的shell為 Bourne Shell

#抓資料並更新
rrd_path="/root/rrd/test.rrd" //這個位址請指定為您存放rrd檔的目錄
rrd_data="192.168.0.1"      //這個IP位址請自行換成您要偵測的主機(目標主機須有執行:snmpd)
image_path="/root/rrd/html" //裡請把""裡面替換成您要存放圖檔的目錄
sec=300
while [ 1 ]              //這裡使用無窮迴圈跑此Script
do
input=`snmpget -c public $rrd_data ifInOctets.4 | sed -e 's/.*ter32: $.*$/\1/'`
output=`snmpget -c public $rrd_data ifOutOctets.4 | sed -e 's/.*ter32: $.*$/\1/'`
now=`date +%s`
echo "rrdtool update $rrd_path $now:$input:$output" >> test.cmd   //紀錄update的資料
rrdtool update $rrd_path $now:$input:$output

#畫圖
rrdtool graph $image_path/test.png \
--title "Local Network" \                   \\換成您想要的圖檔標題
DEF:v1=$rrd_path:input:AVERAGE \
DEF:v2=$rrd_path:output:AVERAGE \
AREA:v1#FF0000:"Input" \
AREA:v2#00FF00:"Output" \
-v "Value"
sleep $sec
done　

可以跑出一個圖，大概如下圖︰

上面只是個極為簡單的範例，事實上還有很多的參數可以設定，可以讓圖表更為精緻，您可以參考官方網站或者是下面的RRDtool完全攻略教學文件獲得進一步的資訊！

RRDtool備份

聰明的您在看完上面的文章一定會想到，如果哪一天您的愛機遭遇不測(被雷打中?!)，那裡頭的寶貴資料不就全毀了? 其實在RRDtool中最重要的資料檔也不過是所謂的RRD資料檔，只要有這個檔圖就可以再畫，所以我們只要利用簡單的Shell Script就能達到備份的效果︰

以下假設您的電腦有兩顆硬碟，以下的範例示範如何定期的把資料備份到另外一個槽的資料夾︰

#!/bin/sh
rrd_path="/root/rrd"        //RRD檔所在資料夾
rrd_file="a.rrd b.rrd c.rrd"    //RRD的檔名，可能不只一個，此例為三個
dest="/home/backup"     //備份檔預定存放的資料夾

for data in $rrd_file
do
cp $rrd_path/$data $dest
done

假設此Script 名稱為 backup.sh 放在/root/rrd資料夾底下，則我們可以在/etc/crontab這個檔或是打"crontab -e"裡面加入以下這行︰

* 12 * * * root /root/rrd/backup.sh //在每天中午十二點備份這些RRD檔到另外一個資料夾

參考資料:
rrdtool完全攻略　出處:酷學園　作者:Abelyang

RRDtool 官方網站﹕http://www.rrdtool.org

Blog of JoJo 2009-02-28 13:12 发表评论

在MySql上实现Replication(Master 与 Slave 数据同步)

Blog of JoJo — Thu, 12 Feb 2009 07:51:00 GMT

摘要: 1：首先确定Master和Slave的数据库版本，Master数据库的版本不能高于Slave数据的版本。这里我是使用MySql 5.0.27 作为Master数据库，MySql 6.0.3（alpha）作为Slave进行测试。 2：首先修改Master数据库的配置文件my.ini (windows), /etc/my.cnf(linux)里 [mysqld] datad... 阅读全文

Blog of JoJo 2009-02-12 15:51 发表评论

正则表达式详解（Perl）

Blog of JoJo — Tue, 03 Feb 2009 07:19:00 GMT

定义：正则表达式是一种字符串模式，可用来和字符串进行匹配。匹配可能成功，也可能失败。许多UNIX命令，包括grep, sed, awk, ed, vi, emacs都有正则表达式的功能。Perl也有这种功能。Python也具有这种功能。

简单示例：

例如：

grep abd readme.file > result.txt

则Perl中，写成

if(/abc/)

{

print $_;

}

在Perl中，正则表达式用斜线标记，以代表斜线之间的内容是个正则表达式。当表达式北斜线包围时，Perl会拿$_和它作比较，以判断条件的真假。

while(<>) {

if (/abc/) {

print $_;

}

这个程序片断能读入某文件的所有内容，并进行匹配。

grep “ab*c” readme.file > result

while( <> )

{

if (/ab*c/) {

print $_;

}

以上程序片断表示以a开头，后面跟0个以上的b,最后以c结尾。

替代运算符以s字母开头，跟正则表达式，再跟替代的运算符号。

s/ab*c/def/

变量是$_会跟正则表达式(ab*c)做比较。进行匹配。

类型：一个“正则表达式”就是一种类型pattern。分为单字符类型和多字符类型。

单字符类型：单字符类型是最常见和最常用的类型。如a。单字符类型”.”能跟除换行符号(\n)的任何字符匹配。如

/a./

会和任何长度为二，且开头是a的字符串匹配。除了”a\n”外。

字符类：(character class)，写法是左右两个中括号([])，内放字符。即字符串对应此类型的地方，若出现唯一一个符合括号中列出的字符，就算符合。

/[abcde]/

/[aeiouAEIOU]/

若在中括号中放[或者]，则需要在前面加反斜杠。如果想表示一段范围，则可以用破折号(dash-)连接。想表示破折号，则前面也必须加反斜杠。

[0123456789]

[0-9]

[0-9\-]

[a-zA-Z]

也可以用“反例”表示字符类别，只要在原来的写法前加上“^”即可。此种类型的意思是任何不在此类别内的单一字符才算上匹配。

[^0-9]

[^aeiouAEIOU]

[^\^]

预定义的字符类别

字符	类别	反例	反例类别
\d	[0-9]	\D	[^0-9]
\w	[a-zA-Z]	\W	[^a-zA-Z]
\s	[\r\t\n\f]	\S	[^\r\t\n\f]

[\da-fA-F] #十六进制数字

类型组合：grouping pattern就是将正则表达式组合起来用。

系列：sequence, 如”abc”

重复符号：multiplier，如星号(*)表示出现零次以上；加号(+)则表示出现一次以上；问号(?)则表示出现零次或一次。

/fo+ba?r/

以上的类型都是贪婪型的greedy。

$_ = “fred xxxxxxxxx barney”;

s/x+/boom/

一般重复符号：(general multiplier)，写法是一对大括号中放一个或两个数字，如/x{5, 10}/，表示x出现5到10次；/x{5,}/表示出现5次以上；/x{5}/表示刚好5个x；/x{0,5}/意思就是”0到5个”

正则表达式如出现两个重复符号，必须遵循“最左边最贪心”(leftmost is greediest)的规则。

$_ = “a xxx c xxxxxxxxxx c xxx d”;

/a.*c.*d/;

以这个例子，第一个“.”就会和第二个c之前所有的字符符合。可以在重复符号后面加上问号，让它变得“不贪心”(nongreedy)。

$_=”a xxx c xxxxxx c xxx d”;

/a.*c?.*d/

把字符串和正则表达式再该一下：

$_=”a xxx ce xxxxxxx ci xxx d”;

/a.*ce.*d/

这个例子中.*如果匹配到第二个c，则e没有办法匹配。故Perl会重试，降低了效率。实际上只要加个”?”就能让Perl少做很多事。

把括号当记忆空间：另外一个组合符号，是前后包围任何类型的括号对(parentheses pair)。括号不会改变类型匹配的情形，不过可以把匹配的部分记录下来，以后参考。调用记住部分，写法是在反斜线后面加上数字。

/fred(.)barney\1/;

会和fredxbarneyx匹配，不会和fredxbarneyy匹配。但是/fred.barney./和两个字符串都匹配。

其中\1代表正则表达式中，第一个用括号括起来的部分。依次有\2, \3…

/a(.)b(.)c\2d\1/;

可以在括号中放入一个以上的字符。如/a(.*)b\1c/;

选项符号：还有一种组合符号称为选项符号(alternation)，如“a|b|c”。选项可以不只一个字符，如/song|blue/。

定位类型：anchor pattern定位类型有好几种。定位类型\，表示目标字符串的这个地方必须是文字边界(word boundary)。所谓“文字边界”就是指符合\w或\W的字符，二个中间的位置；或者界于任何符合\w的字符，与字符串开头（或结尾）中间的位置。

/fred\b/； #会和fred匹配，不会和Frederick匹配

/\bmo/； #会和moe和mole匹配，不会和Elmo匹配

/\bFred\b/； #会和Fred匹配，但Frederick或alFred都不匹配

/\b\+\b/； #会和x+y匹配，不会和++或+匹配

另一个定位类型\B所在之处就不一定是文字边界，如

/\bFred\B/； #会和Frederich匹配，但Fred Flintstone不匹配

还有两种类型，代表字符串的开头或结尾的前一个字符。“^”会和字符串的“开头”匹配。^a会匹配以a开头的字符串，不过a^会和”a”和” ^”这两个字符串匹配。即”^”失去了特别的意义。如果想表示字符串开头是有个^号，则用\^即可。另外一个定位类型是”$”，不过表示字符串结尾。c$表示最后出现的字符是c的那个字符串。表示$这个字符，用\$表示。

优先级：

名称	表示法
括号(parentheses)	() (?:)
重复运算符(multiplier)	? + * {m, n} ?? +? *? {m, n}?
序列和定位符号(sequence and anchoring)	abc^$\A\Z(?=) (?!)
选项符号(alternation)	\|

abc* #会和ab, abc, abcc, abccc, abcccc匹配

(abc)*

^x|y

^(x|y)

a|bc|d

(a|b)(c|d)

(song|blue)bird

=~运算符：如果拿来匹配的字符串，不放在$_变量，则可以用”=~”运算符来解决：可把正则表达式放在运算符右侧，左侧则是想比较的字符串。即把正则表达式的默认目标转向运算符的左边运算单元。

$a = “hello world!”;

$a =~ /^he/;

$a=~/(.)\1/;

if($a=~/(,)\1/)

任何能传回标量字符串值的语句，都可以当作=~运算符的左边单元。如，如

print “any last request?”;

if( =~ /^[yY]/) {

print “And just what might that request be”;

print “Sorry, I’m unable to do that.\n”

}

忽略大小写：grep有-I这个旗标，意思是忽略大小写；在Perl中提供了类似的方法：在正则表达式第二个斜线之后写上I即可。如/somepattern/i。

print “any last request?”;

if ( =~ /^y/i)

{

}

指定界限符号：正则表达式前后用斜线包起来，斜线叫分界符号(delimiter)，如果想在正则表达式表示斜线这个“字符”，在前面加反斜线。

$path =

if($path =~ /^\/usr\/etc/) {

}

Perl允许设计者自行指定分界符合。可用任何－个非字母、非数字、非空白的字符为分界符号，方法是在第一次使用之前写”m”。

/^\/usr\/etc/

m@^/user/etc@

m#^/usr/etc#

内插变量：正则表达式也可以内插变量。

$what = “bird”

$sentence = “Every good bird does fly.”;

if($sentence =~ /\b$what\b/)

{

print “The sentence contains the word $what\n”;

}

下面的例子稍微复杂一些：

$sentence = “Every good bird does fly.”;

print “What should I look for?”;

$what = ;

chomp($what);

if($sentence =~ /$what/)

{

print “I saw $what in $sentence.\n”;

}

else

{

print “nope… didn’t find it.\n”;

}

特别的只读变量：匹配成功后，名为$1, $2, $3…的这些变量，它们的值被分别设为\1,\2,\3…之值。

$_ = “this is a test”;

/(\w+)W+(\w+)/；#和头两个文字匹配；现在$1是”this”，$2是”is”。

也可以在列表环境下，一次取得所有匹配的片断。

$_=”this is a test.”;

($first, $second) = /(\w+)W+(\w+)/;

其他预定义的只读变量包括：”$&”，代表字符串中匹配正则表达式的部分；”$`”代表匹配处之前的部分；”$’”代表匹配处之后的部分。

$_=”this is a sample string.”;

/sa.*le/; #和sample匹配。s`是”this is a “，$&是”sample”，$’是”string”

替代运算符：s/old-regex/new-string/这样的用法是最简单的一种替代运算符。如果想让替代运算符在每个匹配的地方取代，而非仅作用在第一个匹配处，在运算符加g.

$_=”foot fool buffoon”;

s/foo/bar/g;

替代运算符也可以内插变量，像这样：

$_=”hello, world”;

$new = “goodbye”;

s/hello/$new/;

可以在替代运算符里用字符类型，就不会和固定的字符匹配了：

$_=”this is a test”;

s/(\w+) /<$1>/g; #$_现在是

替代运算符可以加上i,代表忽略大小写，如果已经有g了，则i可以出现在g前后。斜线还可以用其他符号代替。s#fred#barney#;替代运算符也可以用=~改变作用的目标。

split和join函数：正则表达式可以把字符串分成许多字段。split函数可以这样做。而join函数则可以把这些片断组合起来。 split函数有两个参数，分别是正则表达式和字符串，它会寻找字符串中匹配正则表达式的部分，其他不匹配的部分会依序用列表值方式传回。

$line= “merlyn::118:10:Randal:/home/merlyn:/urs/bin/perl “;

@fields = split(/:/, $line);

如果要匹配一个以上的:则用：

@fields = split(/:+, $line);

变量$_是split函数的第二个参数的默认值：

$_=”some string”;

@words =split(/ /);

由于参数字符串内没有连续的空白字符，结果会是空字符串。理想的写法是/ +/, /\s+/和”一个以上“的空白字符匹配。这是第一个参数的默认值。

@words = split;用空白字符分割$_。

$line=”merlyn::118:10:Randa;:/home/merlyn:”);

($name,$password,$uid,$gid,$gcos,$home,$shell)=split(/:/,$line);

join函数跟split函数对应，该函数用第一个参数字符串当作”胶”，把第二个参数里面的值一个个“粘“起来。

$bigstring=join($glue,@list);

Blog of JoJo 2009-02-03 15:19 发表评论

Synchronizing Large Applications

Blog of JoJo — Thu, 22 Jan 2009 04:17:00 GMT

Synchronizing Large Applications

When your environment contains large applications to synchronize or available memory is constrained, you can adjust the JVM options to limit memory usage. This adjustment reduces the possibility of receiving out of memory errors. The instance synchronization JVM uses default settings, but you can configure JVM options to change them.

Set the JVM options using the INSTANCE-SYNC-JVM-OPTIONS property. The command to set the property is:

asadmin set 

domain.node-agent.node_agent_name.property.INSTANCE-SYNC-JVM-OPTIONS="JVM_options"

For example:

asadmin set 

domain.node-agent.node0.property.INSTANCE-SYNC-JVM-OPTIONS="-Xmx32m -Xss2m"

In this example, the node agent is node0 and the JVM options are -Xmx32m -Xss2m.

For more information, see http://java.sun.com/docs/hotspot/VMOptions.html.

Another example:

bin/asadmin set domain.node-agent.agent2.property.INSTANCE-SYNC-JVM-OPTIONS="-Xmx512m"

Note –

Restart the node agent after changing the INSTANCE-SYNC-JVM-OPTIONS property, because the node agent is not automatically synchronized when a property is added or changed in its configuration.

Using the doNotRemoveList Flag

If your application requires to store and read files in the directories (applications, generated, docroot, config, lib, java-web-start) that are synchronized by the Application Server, use the doNotRemoveList flag. This attribute takes a coma-separated list of files or directories. Your application dependent files are not removed during server startup, even if they do not exist in the central repository managed by DAS. If the same file exists in the central repository, they will be over written during synchronization.

Use the INSTANCE-SYNC-JVM-OPTIONS property to pass in the doNotRemoveList attribute.

For example:

...

Blog of JoJo 2009-01-22 12:17 发表评论

How to Automate Secure File Synchronization using SSH and rsync

Blog of JoJo — Tue, 20 Jan 2009 09:18:00 GMT

How to Automate Secure File Synchronization using SSH and rsync

Tom Hilinski
Natural Resource Ecology Laboratory,
Colorado State University
Last updated: Dec 2008

Introduction

In order to automate file transfers between computers without a password, a private-public key identification key simplifies the process. This is useful, for instance, when using rsync to synchronize files in local and remote directories. For instance, after editing files on your local Linux desktop or Microsoft Windows laptop, you want to automatically update the files on the office computer with your modified files. In this case, a utility such as rsync can be used to do the update without prompting you for a password on the office computer.

The process of creating the key is described here in the context of using rsync on a local Windows computer with Cygwin installed. Use the Cygwin setup program to install SSH, rsync, and Bash. Here, I assume that the remote computer, say, your office server, is running Linux, and you have an account on it with the user name yourUserName.

In the examples below, command lines begin with a $ character, comment lines begin with a # character, while text beginning without either is written to the console display. Example text that is italicized means you substitute your own information there; for example, yourUserName is replace by your actual user name.

Create a Key

On your local Windows computer, open a Bash shell console window. If you don't have a directory named .ssh create one by using SSH to connect to your office computer for the first time. You will be prompted to accept the key.

$ ls -d .ssh

# if the directory does not exist, run ssh

$ ssh yourUserName@calypso.nrel.colostate.edu

Go into the .ssh directory and create a key. This key will have two files, a private file and a public file. When prompted, do not enter a password or passphrase.

$ ssh-keygen -t dsa -b 1024 -f yourUserName-rsync-key

Generating public/private dsa key pair.

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in yourUserName-rsync-key.

Your public key has been saved in yourUserName-rsync-key.pub.

The key fingerprint is:

(a long string of hexadecimal digits)

Check the permissions on your key files (e.g., ls -l). The permissions should be 600 (or rw-----).

On your office computer, make sure you have in your home directory, a subdirectory named .ssh (note the leading dot).

$ ssh yourUserName@calypso.nrel.colostate.edu

$ ls -d .ssh

# If this directory doesn't exist, create it:

$ mkdir .ssh

# Make sure the permissions are secure:

$ chmod 700 .ssh

Now, log off your office computer.

Next, copy the public key file to your office computer, log onto that computer, then append the key file to the SSH file containing keys it knows about.

# Copy the public key to your office computer:

$ scp yourUserName-rsync-key.pub yourUserName@calypso.nrel.colostate.edu:/home/nrel/yourUserName/.ssh/

# Log on to the remote computer:

$ ssh yourUserName@calypso.nrel.colostate.edu

# If your are not in a bash shell, then start one:

$ bash

# If the key file does not exist, create it:

$ if [ ! -f authorized_keys ]; then touch authorized_keys; chmod 600 authorized_keys; fi

# Append your new public key to the key file:

$ cat yourUserName-rsync-key.pub >> authorized_keys

$ rm yourUserName-rsync-key.pub

Your key is now ready to use with rsync. Optionally, you can restrict the use of the key to an IP address and a particular process (e.g., rsync). To restrict the key to rsync, create the file listed in Appendix A in your ~/.ssh directory on your office computer. You can use a text editor to paste that text in. Then set the permissions so no one else can read it. For example:

# Use vi to create the file; paste in the script from Appendix A.

$ vi restrict-to-rsync

$ chmod 700 restrict-to-rsync

Next, edit the file authorized_keys so that the line with your key (the last line, since the key was just appended to the file) begins with a command to run that script. The command points to the full path of the script file. The line originally began with:

ssh-dss AAAAB3...

After inserting the script command, the line starts with:

command="/home/nrel/yourUserName/.ssh/restrict-to-rsync" ssh-dss AAAAB3...

Using rsync With SSH and Your Key

Test the use of your new key by copying a junk file from your local computer to your office computer. Here, the local file is junk.txt and the remote directory in your office computer is tmp, and the direction of transfer is local-to-remote. Give SSH the name of your private key file on your local computer, including its path, using the following form:

rsync -auvz -e "ssh -i private-key-file" source destination

Here, source is a file or a directory, and destination has the form yourUserName@remote-computer:/remote-path

A real example, using the file names from the previous examples, is:

rsync -auvz -e "ssh -i /home/yourUserName/.ssh/yourUserName-rsync-key" junk.txt yourUserName@calypso.nrel.colostate.edu

The rsync flags -auvz specify "archive", "update", "verbose messages", and "compress files for transfer", respectively. "Update" means that files on the destination that are newer than your local files are not overwritten. The "-e" flag tells rsync the SSH command.

If you want details on what SSH is doing, add "-v" to the ssh options. To run rsync quietly, remove the "-v" option from both rsync and SSH option list.

To reverse the synchronization so the remote file is updated on your local computer, reverse the source and destinations.

You can store your rsync commands that you use all the time in a script file. Keep this script with your project files or in a script directory that is specified in your PATH environment variable.

Additional Information

rsync document: http://rsync.samba.org/ftp/rsync/rsync.html

rsync web site: http://rsync.samba.org/

Acknowledgements

Many online sources provided the information I used to create this process. A particularly succinct source was http://troy.jdmz.net/rsync/index.html provided the basis of the script in Appendix A. Thanks to all.

Appendix A: File `restrict-to-rsync`

The following shell script checks that rsync is the process attempting to connect. If it is not, the script fails, and SSH also fails. A log file named validate-rsync.log is created or appended to with each connection.

#!/bin/sh

logfile=/home/nrel/yourUserName/.ssh/restrict-to-rsync.log

case "$SSH_ORIGINAL_COMMAND" in

*\&*)

echo `date` "- SSH connection rejected" >> $logfile

;;

*\(*)

echo `date` "- SSH connection rejected" >> $logfile

;;

*\{*)

echo `date` "- SSH connection rejected" >> $logfile

;;

*\;*)

echo `date` "- SSH connection rejected" >> $logfile

;;

*\<*)

echo `date` "- SSH connection rejected" >> $logfile

;;

*\`*)

echo `date` "- SSH connection rejected" >> $logfile

;;

*\|*)

echo `date` "- SSH connection rejected" >> $logfile

;;

rsync\ --server*)

{

echo `date` "- SSH connection accepted" >> $logfile

$SSH_ORIGINAL_COMMAND

}

;;

*)

echo `date` "- SSH connection rejected" >> $logfile

;;

esac

Blog of JoJo 2009-01-20 17:18 发表评论

Remote Backups With Rsync

Blog of JoJo — Tue, 20 Jan 2009 08:29:00 GMT

Remote Backups With Rsync

In this article we explain how to automate the backup of files on remote machines to a centralized server using rsync.

rsync is a command line utility that is used to synchronize files between two computers over a network to synchronize files between two filesystems. It was written as a replacement for rcp but with many new features. For example it uses an algorithm that will only transfer files that have been modified. SSH will be used to authenticate between the machines and to encrypt the network traffic.

The situation: We have four machines named: server, machine1, machine2, and machine3. The server has a tape drive that is used to do nightly backups. machine1 is used as a development box and has files that need to be backed up in /src and in /home. machine2 is used for mail and needs /home and /mail to be backed up. machine3 is a web server and needs /home, /var/www, and /etc/httpd backed up.

Create a shell script for each machine. Simplify your maintenance by placing the scripts in a central location. I like to use /root/scripts. Decide on where you want to log your output. I like /root/logs but another common option is to have the script mail you the output.

Add entries to your crontab to call the scripts. Make sure you leave enough time before your normal backups of the server that the rsync jobs complete.

Each night the following will occur:

rsync machine1 -> Server
rsync machine2 -> Server
rsync machine3 -> Server
backup server to tape

Let's take a look at the flags used for rsync in the examples:

rsync -ave ssh --numeric-ids --delete machine1:/home /machine1

-a:
Archive mode
-v:
Verbose output
-e ssh:
Specify the remote shell as ssh
--numeric-ids:
Tells rsync to not map user and group id numbers local user and group names
--delete:
Makes server copy an exact copy of the source by removing any files that have been removed on the remote machine
machine1:/home:
The remote machine name, then the directory to be backed up
/machine1:
The directory to place the backup

Next generate a public private key pair with ssh. Place the public key in the ~/.ssh/authorized_keys file in an account on machine1, machine2, and machine3 that has read access to the directories that need to be backed up. It is best not to use the root account on the remote machines, but you should evaluate the risk in your environment. Test that you can login to these accounts using ssh without using a password.

Test each one of the rsync scripts. The first time you run rsync will take the longest as it will need to copy all the files from the remote machines and not just the files that have changed.

Add the /machine1, /machine2, and /machine3 (or whatever you have named them) directories to the servers backup script.

While this process does not backup the entire remote machine, it will ensure that you will not lose irreplaceable data.

Starting with the example scripts included in this tutorial there are many changes that can be made to fit your specific circumstances.

The frequency of the rsyncs can be modified to occur more often or at different times. Simply by adding additional crontab lines the backup from the remote machines could be done everyday at lunch, multiple times a day or even hourly.

The scripts could also be changed to rotate between multiple backups on the server or could be changed to do some sort of processing on the files before they are backed up. For example if the home directories you are backing up contain web browser caches, they could be removed after the rsync but before the system backup.

Using this article as a starting point you should create a backup plan that fit your needs.

Example rsync script for machine1:

#!/bin/bash



rsync -ave ssh --numeric-ids --delete machine1:/home /machine1

rsync -ave ssh --numeric-ids --delete machine1:/src /machine1

Example rsync script for machine2:

#!/bin/bash



rsync -ave ssh --numeric-ids --delete machine2:/home /machine2

rsync -ave ssh --numeric-ids --delete machine2:/mail /machine2

Example rsync script for machine3:

#!/bin/bash



rsync -ave ssh --numeric-ids --delete machine3:/home /machine3

rsync -ave ssh --numeric-ids --delete machine3:/var/www /machine3

rsync -ave ssh --numeric-ids --delete machine3:/etc/httpd /machine3

Example crontab file logging to a directory:

# Scripts to rsync machines

59 20 * * * /root/scripts/sync-machine1.sh >/root/logs/sync-machine1.log 2>&1

59 21 * * * /root/scripts/sync-machine2.sh >/root/logs/sync-machine2.log 2>&1

59 22 * * * /root/scripts/sync-machine3.sh >/root/logs/sync-machine3.log 2>&1

#

# Nightly Backup script

59 23  * * * /root/scripts/backup.sh > /root/logs/backup.log 2>&1

Example crontab file mailing the output:

# Scripts to rsync machines

59 20 * * * /root/scripts/sync-machine1.sh

59 21 * * * /root/scripts/sync-machine2.sh

59 22 * * * /root/scripts/sync-machine3.sh

#

# Nightly Backup script

59 23  * * * /root/scripts/backup.sh

Blog of JoJo 2009-01-20 16:29 发表评论

PHP Requirements

Blog of JoJo — Wed, 14 Jan 2009 03:25:00 GMT

PHP Requirements

As PHP version the eZ Components require version 5.2.1, although a later version is always recommended. The latest version of PHP can be downloaded here.

eZ Components versions before the 2008.1 release require PHP version 5.1.6. Please be aware that no updates for those releases are being made anymore.

Some of the components require that certain PHP extensions are enabled in the build. While most of them are enabled by default, some of them have to added explicitly. The table below lists all the components, and which extensions are required or optional.

Required and Preferred Extensions

Component	Extension (default)	Extension Required	Extension Optional
All	pcre, spl, reflection
Authentication			openssl and bcmath or gmp for TypeKey and OpenID support, ldap for LDAP support
AuthenticationDatabaseTiein			Same as Database
Archive	zlib, posix	bz2	posix for permission support
Base	posix		posix to test if features are enabled
Cache			memcache and zlib for Memcache storage, apc for APC storage
Configuration	posix
ConsoleTools
Database	pdo, pdo_sqlite		pdo_mysql, pdo_oci8, pdo_pgsql, pdo_mssql/pdo_dblib, depending on the database that you want to use
DatabaseSchema	simplexml		Same as Database
Debug
EventLog			Same as Database, if you want to log to a database
Execution
Feed	dom
File
Graph	dom, xml		gd to generate bitmaps (with TrueType font support or Type1 font support), ming for generating Graphs as flash files
GraphDatabaseTiein	pdo		Same as Database
ImageAnalysis	ctype		exif
ImageConversion			gd
Mail	iconv		fileinfo for better detection of mime-types when sending e-mails, mcrypt for NTLM SMTP authentication, openssl for SSL support
PersistentObject
PhpGenerator
SignalSlot
SystemInformation			win32ps (on Windows)
Template
Translation	ctype, simplexml
TranslationCacheTiein
Tree	dom
TreeDatabaseTiein	pdo		Same as Database
Url
UserInput	filter
Webdav			fileinfo or mimetype for detection of mime-types of files
Workflow	simplexml, dom
WorkflowDatabaseTiein			Same as Database
WorkflowEventLogTiein			Same as Database, if you want to log to a database

Quick Reference on Enabling Extensions

Extensions that are enabled by default in PHP

Extension	Configure switch(es)
ctype	--enable-ctype
dom	--enable-dom --enable-libxml
iconv	--with-iconv
pcre	--with-pcre-regex
pdo	--enable-pdo
pdo_sqlite	--with-pdo-sqlite
posix	--enable-posix
reflection	--enable-reflection
simplexml	--enable-simplexml
spl	--enable-spl
xml	--enable-xml
zlib	--with-zlib

Extensions that are bundled with PHP

Extension	Configure switch(es)	Remarks
bcmath	--enable-bcmath
bz2	--with-bz2
gd	--with-gd --with-freetype-dir=/usr --with-jpeg-dir=/usr --with-ttf --with-xpm-dir --with-png-dir --with-jpeg-dir --with-t1lib	The eZ Components want at least PNG and JPG support, other parts are optional. We suggest to compile GD also with FreeType2 support as well to enable all features of the Components.
gmp	--with-gmp
ldap	--with-ldap
mcrypt	--with-mcrypt
mimetype	--with-mime-magic	Deprecated in favour of fileinfo.
ming	--with-ming
exif	--enable-exif
filter	--enable-filter	In PHP 5.2.0 and higher you can simply use --enable-filter, in PHP 5.1.x you need to install the filter extension through PECL.
pdo_mysql	--with-pdo-mysql
pdo_oci8	--with-pdo-oci
pdo_pgsql	--with-pdo-pgsql
pdo_mssql	--with-pdo-mssql	On Windows (on Linux use pdo_dblib)
pdo_dblib	--with-pdo-dblib	Mssql driver for Linux (on Windows use pdo_mssql). The FreeTDS software must be installed also.
openssl	--with-openssl

Extensions that are not bundled with PHP

Extension Configure switch(es)

apc You need to install the APC extension through PECL.

Extension	Configure switch(es)
apc	You need to install the APC extension through PECL.
fileinfo	You need to install the fileinfo extension through PECL. On Windows additional configuration steps are required: download `magic` and `magic.mime` files from gnuwin32 (from the Binaries Zip in the File for Windows package) and copy them to a folder, for example the folder where PHP is installed (e.g. `c:\php`) in System settings, setup the enviroment variable MAGIC to point to the folder where you copied the magic files, plus the `magic` part (e.g. `c:\php\magic`). Restart Windows if needed. alternatively you can use this command-line before running your PHP scripts: set MAGIC=c:\php\magic
memcache	You need to install the Memcache extension through PECL.

fileinfo

You need to install the fileinfo extension through PECL.

On Windows additional configuration steps are required:

download magic and magic.mime files from gnuwin32 (from the Binaries Zip in the File for Windows package) and copy them to a folder, for example the folder where PHP is installed (e.g. c:\php)
in System settings, setup the enviroment variable MAGIC to point to the folder where you copied the magic files, plus the magic part (e.g. c:\php\magic). Restart Windows if needed.
alternatively you can use this command-line before running your PHP scripts:
```
set MAGIC=c:\php\magic

                
```

memcache You need to install the Memcache extension through PECL.

Blog of JoJo 2009-01-14 11:25 发表评论

[转载]Ubuntu/Debian包管理命令大全（apt & dpkg）

Blog of JoJo — Fri, 09 Jan 2009 03:57:00 GMT

[转载]Ubuntu/Debian包管理命令大全（apt & dpkg）

apt-cache search # ------(package 搜索包)
apt-cache show #------(package 获取包的相关信息，如说明、大小、版本等)
apt-get install # ------(package 安装包)
apt-get install # -----(package --reinstall 重新安装包)
apt-get -f install # -----(强制安装, "-f = --fix-missing"当是修复安装吧...)
apt-get remove #-----(package 删除包)
apt-get remove --purge # ------(package 删除包，包括删除配置文件等)
apt-get autoremove --purge # ----(package 删除包及其依赖的软件包+配置文件等（只对6.10有效，强烈推荐）)
apt-get update #------更新源
apt-get upgrade #------更新已安装的包
apt-get dist-upgrade # ---------升级系统
apt-get dselect-upgrade #------使用 dselect 升级
apt-cache depends #-------(package 了解使用依赖)
apt-cache rdepends # ------(package 了解某个具体的依赖,当是查看该包被哪些包依赖吧...)
apt-get build-dep # ------(package 安装相关的编译环境)
apt-get source #------(package 下载该包的源代码)
apt-get clean && apt-get autoclean # --------清理下载文件的存档 && 只清理过时的包
apt-get check #-------检查是否有损坏的依赖
dpkg -S filename -----查找filename属于哪个软件包
apt-file search filename -----查找filename属于哪个软件包
apt-file list packagename -----列出软件包的内容
apt-file update --更新apt-file的数据库

dpkg --info "软件包名" --列出软件包解包后的包名称.
dpkg -l --列出当前系统中所有的包.可以和参数less一起使用在分屏查看. (类似于rpm -qa)
dpkg -l |grep -i "软件包名" --查看系统中与"软件包名"相关联的包.
dpkg -s 查询已安装的包的详细信息.
dpkg -L 查询系统中已安装的软件包所安装的位置. (类似于rpm -ql)
dpkg -S 查询系统中某个文件属于哪个软件包. (类似于rpm -qf)
dpkg -I 查询deb包的详细信息,在一个软件包下载到本地之后看看用不用安装(看一下呗).
dpkg -i 手动安装软件包(这个命令并不能解决软件包之前的依赖性问题),如果在安装某一个软件包的时候遇到了软件依赖的问题,可以用apt-get -f install在解决信赖性这个问题.
dpkg -r 卸载软件包.不是完全的卸载,它的配置文件还存在.
dpkg -P 全部卸载(但是还是不能解决软件包的依赖性的问题)
dpkg -reconfigure 重新配置

apt-get install
下载软件包，以及所有依赖的包，同时进行包的安装或升级。如果某个包被设置了 hold (停止标志，就会被搁在一边(即不会被升级)。更多 hold 细节请看下面。
apt-get remove [--purge]
移除以及任何依赖这个包的其它包。
--purge 指明这个包应该被完全清除 (purged) ，更多信息请看 dpkg -P。

apt-get update
升级来自 Debian 镜像的包列表，如果你想安装当天的任何软件，至少每天运行一次，而且每次修改了
/etc/apt/sources.list 後，必须执行。

apt-get upgrade [-u]
升级所有已经安装的包为最新可用版本。不会安装新的或移除老的包。如果一个包改变了依赖关系而需要安装一个新的包，那么它将不会被升级，而是标志为 hold。apt-get update 不会升级被标志为 hold 的包 (这个也就是 hold 的意思)。请看下文如何手动设置包为 hold。我建议同时使用 '-u' 选项，因为这样你就能看到哪些包将会被升级。

apt-get dist-upgrade [-u]
和 apt-get upgrade 类似，除了 dist-upgrade 会安装和移除包来满足依赖关系。因此具有一定的危险性。

apt-cache search
在软件包名称和描述中，搜索包含xxx的软件包。

apt-cache show
显示某个软件包的完整的描述。

apt-cache showpkg
显示软件包更多细节，以及和其它包的关系。

dselect
console-apt
aptitude
gnome-apt
APT 的几个图形前端(其中一些在使用前得先安装)。这里 dselect 无疑是最强大的，也是最古老，最难驾驭。

普通 Dpkg 用法
dpkg -i
安装一个 Debian 包文件，如你手动下载的文件。

dpkg -c
列出的内容。

dpkg -I
从中提取包信息。

dpkg -r
移除一个已安装的包。

dpkg -P
完全清除一个已安装的包。和 remove 不同的是，remove 只是删掉数据和可执行文件，purge 另外还删除所有的配制文件。

dpkg -L
列出安装的所有文件清单。同时请看 dpkg -c 来检查一个 .deb 文件的内容。

dpkg -s
显示已安装包的信息。同时请看 apt-cache 显示 Debian 存档中的包信息，以及 dpkg -I 来显示从一个 .deb 文件中提取的包信息。

dpkg-reconfigure
重新配制一个已经安装的包，如果它使用的是 debconf (debconf 为包安装提供了一个统一的配制界面)。你能够重新配制 debconf 它本身，如你想改变它的前端或提问的优先权。例如，重新配制 debconf，使用一个 dialog 前端，简单运行：

dpkg-reconfigure --frontend=dialog debconf (如果你安装时选错了，这里可以改回来哟：)

echo " hold" | dpkg --set-selections
设置的状态为 hlod (命令行方式)

dpkg --get-selections ""
取的的当前状态 (命令行方式)

支持通配符，如：
Debian:~# dpkg --get-selections *wine*
libwine hold
libwine-alsa hold
libwine-arts hold
libwine-dev hold
libwine-nas hold
libwine-print hold
libwine-twain hold
wine hold
wine+ hold
wine-doc hold
wine-utils hold

例如：
大家现在用的都是 gaim-0.58 + QQ-plugin，为了防止 gaim 被升级，我们可以采用如下方法：

方法一：
Debian:~# echo "gaim hold" | dpkg --set-selections
然後用下面命令检查一下：
Debian:~# dpkg --get-selections "gaim"
gaim hold
现在的状态标志是 hold，就不能被升级了。

如果想恢复怎么办呢?
Debian:~# echo "gaim install" | dpkg --set-selections
Debian:~# dpkg --get-selections "gaim"
gaim install
这时状态标志又被重置为 install，可以继续升级了。

同志们会问，哪个这些状态标志都写在哪个文件中呢?
在 /var/lib/dpkg/status 里，你也可以通过修改这个文件实现 hold。

有时你会发现有的软件状态标志是 purge，不要奇怪。
如：事先已经安装了 amsn，然後把它卸了。
apt-get remove --purge amsn
那么状态标志就从 install 变成 purge。

方法二：
在/etc/apt 下手动建一个 preferences 文件
内容：
Package: gaim
Pin: version 0.58*
保存

dpkg -S
在包数据库中查找，并告诉你哪个包包含了这个文件。(注：查找的是事先已经安装的包)

--------------------------------------------
Debian的软件包管理工具命令不完全列表
--------------------------------------------
Debian系统中所有的包信息都在/var/lib/dpkg下.其中/var/lib/dpkg/info目录中保存了各个软件包的信息及管理文件.每个文件的作用如下:
以 ".conffiles"     结尾的文件记录软件包的配置列表.
以 ".list"          结尾的文件记录了软件包的文件列表,用户可在文件当中找到软件包文件的具体安装位置.
以 ".md5sums"       结尾的文件记录了md5信息,用来进行包的验证的.
以 ".config"        结尾的文件是软件包的安装配置角本.
以 ".postinst"      角本是完成Debian包解开之后的配置工作,通常用来执行所安装软件包相关的命令和服务的重新启动.
以 ".preinst"       角本在Debain解包之前运行,主要作用是是停止作用于即将升级的软件包服务直到软件包安装或和升级完成.
以 ".prerm"         脚本负责停止与软件包关联的daemon服务,在删除软件包关联文件之前执行.
以 ".postrm"        脚本负责修改软件包链接或文件关联,或删除由它创建的文件.

/var/lib/dpkg/available是软件包的描述信息.
包括当前系统中所有使用的Debian安装源中所有的软件包,还包括当前系统中已经安装和未安装的软件包.

1.dpkg包管理工具
dpkg --info "软件包名" --列出软件包解包后的包名称.
dpkg -l     --列出当前系统中所有的包.可以和参数less一起使用在分屏查看.
dpkg -l |grep -i "软件包名" --查看系统中与"软件包名"相关联的包.
dpkg -s   查询已安装的包的详细信息.
dpkg -L   查询系统中已安装的软件包所安装的位置.
dpkg -S   查询系统中某个文件属于哪个软件包.
dpkg -I   查询deb包的详细信息,在一个软件包下载到本地之后看看用不用安装(看一下呗).
dpkg -i 手动安装软件包(这个命令并不能解决软件包之前的依赖性问题),如果在安装某一个软件包的时候遇到了软件依赖的问题,可以用apt-get -f install在解决信赖性这个问题.
dpkg -r 卸载软件包.不是完全的卸载,它的配置文件还存在.
dpkg -P 全部卸载(但是还是不能解决软件包的依赖性的问题)
dpkg -reconfigure 重新配置
2. apt高级包管理工具
   (1)GTK图形的"synaptic",这是APT的前端工具.
   (2)"aptitude",这也是APT的前端工具.
   用APT管理工具进行包的管理,可以有以下几种方法做源:
   (1)拿安装盘做源,方法如下:
        apt-cdrom ident        扫描光盘的信息
        apt-cdrom add          添加光盘源
   (2)这也是最常用的方法就是把源添加到/etc/apt/source.list中,之后更新列apt-get update
APT管理工具常用命令
apt-cache 加上不同的子命令和参数的使用可以实现查找,显示软件,包信息及包信赖关系等功能.
apt-cache stats 显示当前系统所有使用的Debain数据源的统计信息.
apt-cache search +"包名",可以查找相关的软件包.
apt-cache show   +"包名",可以显示指定软件包的详细信息.
apt-cache depends +"包名",可以查找软件包的依赖关系.
apt-get upgrade   更新系统中所有的包到最新版
apt-get install   安装软件包
apt-get --reindtall install 重新安装软件包
apt-get remove 卸载软件包
apt-get --purge remove 完全卸载软件包
apt-get clean 清除无用的软件包
在用命令apt-get install之前,是先将软件包下载到/var/cache/apt/archives中,之后再进行安装的.所以我们可以用apt-get clean清除/var/cache/apt/archives目录中的软件包.
源码包安装
   apt-cache showsrc 查找看源码包的文件信息(在下载之前)
   apt-get source 下载源码包.
   apt-get build-dep +"包名" 构建源码包的编译环境.

Blog of JoJo 2009-01-09 11:57 发表评论

Linux_Users_and_Sudo

Blog of JoJo — Wed, 19 Nov 2008 11:18:00 GMT

Introduction

Before we proceed, it would be best to cover some basic user administration topics that will be very useful in later chapters. Adding Users

One of the most important activities in administering a Linux box is the addition of users. Here you'll find some simple examples to provide a foundation for future chapters. It is not intended to be comprehensive, but is a good memory refresher. You can use the command man useradd to get the help pages on adding users with the useradd command or the man usermod to become more familiar with modifying users with the usermod command.

Who Is the Super User?

The super user with unrestricted access to all system resources and files in Linux is the user named root. This user has a user ID, of 0 which is universally identified by Linux applications as belonging to a user with supreme privileges. You will need to log in as user root to add new users to your Linux server.

Debian Note: When installing Ubuntu Linux systems, you are prompted to create a primary user that is not root. A root user is created but no password is set, so you initially cannot log in as this user. The primary user can become the root user using the sudo su - command that will be discussed later.

How To Add Users

Adding users takes some planning; read through these steps below before starting:

1) Arrange your list of users into groups by function. In this example there are three groups "parents", "children" and "soho".

Parents    Children    Soho



Paul       Alice       Accounts

Jane       Derek       Sales

2) Add the Linux groups to your server:

[root@bigboy tmp]# groupadd parents

[root@bigboy tmp]# groupadd children

[root@bigboy tmp]# groupadd soho

3) Add the Linux users and assign them to their respective groups

[root@bigboy tmp]# useradd -g parents paul

[root@bigboy tmp]# useradd -g parents jane

[root@bigboy tmp]# useradd -g children derek

[root@bigboy tmp]# useradd -g children alice

[root@bigboy tmp]# useradd -g soho accounts

[root@bigboy tmp]# useradd -g soho sales

If you don't specify the group with the -g, RedHat/Fedora Linux creates a group with the same name as the user you just created; this is also known as the User Private Group Scheme. When each new user first logs in, they are prompted for their new permanent password.

4) Each user's personal directory is placed in the /home directory. The directory name will be the same as their user name.

 [root@bigboy tmp]# ll /home

drwxr-xr-x    2 root     root        12288 Jul 24 20:04 lost+found

drwx------    2 accounts soho         1024 Jul 24 20:33 accounts

drwx------    2 alice    children     1024 Jul 24 20:33 alice

drwx------    2 derek    children     1024 Jul 24 20:33 derek

drwx------    2 jane     parents      1024 Jul 24 20:33 jane

drwx------    2 paul     parents      1024 Jul 24 20:33 paul

drwx------    2 sales    soho         1024 Jul 24 20:33 sales

[root@bigboy tmp]#

How to Change Passwords

You need to create passwords for each account. This is done with the passwd command. You are prompted once for your old password and twice for the new one.

User root changing the password for user paul.

[root@bigboy root]# passwd paul

Changing password for user paul.

New password: 

Retype new password: 

passwd: all authentication tokens updated successfully.

[root@bigboy root]#

Users might wish to change their passwords at a future date. Here is how unprivileged user paul would change his own password.

[paul@bigboy paul]$ passwd

Changing password for paul

Old password: your current password

Enter the new password (minimum of 5, maximum of 8 characters)

Please use a combination of upper and lower case letters and numbers.

New password: your new password

Re-enter new password: your new password

Password changed.

[paul@bigboy paul]$

How to Delete Users

The userdel command is used to remove the user's record from the /etc/passwd and /etc/shadow used in the login process. The command has a single argument, the username.

[root@bigboy tmp]# userdel paul

There is also an optional -r switch that additionally removes all the contents of the user's home directory. Use this option with care. The data in a user's directory can often be important even after the person has left your company.

[root@bigboy tmp]# userdel -r paul

How to Tell the Groups to Which a User Belongs

Use the groups command with the username as the argument.

[root@bigboy root]# groups paul

paul : parents

[root@bigboy root]#

How to Change the Ownership of a File

You can change the ownership of a file with the chown command. The first argument is the desired username and group ownership for the file separated by a colon (:) followed by the filename. In the next example we change the ownership of the file named text.txt from being owned by user root and group root to being owned by user testuser in the group users:

[root@bigboy tmp]# ll test.txt

-rw-r--r--  1 root root 0 Nov 17 22:14 test.txt

[root@bigboy tmp]# chown testuser:users test.txt

[root@bigboy tmp]# ll test.txt

-rw-r--r--  1 testuser users 0 Nov 17 22:14 test.txt

[root@bigboy tmp]#

You can also use the chown command with the -r switch for it to doe recursive searches down into directories to change permissions.

Using sudo

If a server needs to be administered by a number of people it is normally not a good idea for them all to use the root account. This is because it becomes difficult to determine exactly who did what, when and where if everyone logs in with the same credentials. The sudo utility was designed to overcome this difficulty.

The sudo utility allows users defined in the /etc/sudoers configuration file to have temporary access to run commands they would not normally be able to due to file permission restrictions. The commands can be run as user "root" or as any other user defined in the /etc/sudoers configuration file.

The privileged command you want to run must first begin with the word sudo followed by the command's regular syntax. When running the command with the sudo prefix, you will be prompted for your regular password before it is executed. You may run other privileged commands using sudo within a five-minute period without being re-prompted for a password. All commands run as sudo are logged in the log file /var/log/messages.

Simple Sudo Examples

Using sudo is relatively simple as we can see from these examples.

Temporarily Gaining root Privileges

In this example, user bob attempts to view the contents of the /etc/sudoers file, which is an action that normally requires privileged access. Without sudo, the command fails:

[bob@bigboy bob]$ more /etc/sudoers

/etc/sudoers: Permission denied

[bob@bigboy bob]$

Bob tries again using sudo and his regular user password and is successful:

[bob@bigboy bob]$ sudo more /etc/sudoers

Password:

...

...

[bob@bigboy bob]$

The details of configuring and installing sudo are covered in later sections.

Becoming root for a Complete Login Session

The su command allows a regular user to become the system's root user if they know the root password. A user with sudo rights to use the su command can become root, but they only need to know their own password, not that of root as seen here.

someuser@u-bigboy:~$ sudo su -

Password:

root@u-bigboy:~#

Some systems administrators will use sudo to grant root privileges to their own personal user account without the need to provide a password.

Later sections describe how to disable sudo su ability and also how to use sudo without password prompts.

Downloading and Installing the sudo Package

Fortunately the package is installed by default by RedHat/Fedora which eliminates the need to anything more in this regard. The visudo Command

The visudo command is a text editor that mimics the vi editor that is used to edit the /etc/sudoers configuration file. It is not recommended that you use any other editor to modify your sudo parameters because the sudoers file isn't located in the same directory on all versions of Linux. visudo uses the same commands as the vi text editor. The visudo command must run as user root and should have no arguments:

[root@aqua tmp]# visudo

The /etc/sudoers File

The /etc/sudoers file contains all the configuration and permission parameters needed for sudo to work. There are a number of guidelines that need to be followed when editing it with visudo. General /etc/sudoers Guidelines

The /etc/sudoers file has the general format shown in Table 9-1.

Table 9-1 Format of the /etc/sudoers File

General sudoers File Record Format

usernames/group servername = (usernames command can be run as) command

There are some general guidelines when editing this file:

Groups are the same as user groups and are differentiated from regular users by a % at the beginning. The Linux user group "users" would be represented by %users.
You can have multiple usernames per line separated by commas.
Multiple commands also can be separated by commas. Spaces are considered part of the command.
The keyword ALL can mean all usernames, groups, commands and servers.
If you run out of space on a line, you can end it with a back slash (\) and continue on the next line.
sudo assumes that the sudoers file will be used network wide, and therefore offers the option to specify the names of servers which will be using it in the servername position in Table 9-1. In most cases, the file is used by only one server and the keyword ALL suffices for the server name.
The NOPASSWD keyword provides access without prompting for your password.

Simple /etc/sudoers Examples

This section presents some simple examples of how to do many commonly required tasks using the sudo utility.

Granting All Access to Specific Users

You can grant users bob and bunny full access to all privileged commands, with this sudoers entry.

bob, bunny  ALL=(ALL) ALL

This is generally not a good idea because this allows bob and bunny to use the su command to grant themselves permanent root privileges thereby bypassing the command logging features of sudo. The example on using aliases in the sudoers file shows how to eliminate this prob

Granting Access To Specific Users To Specific Files

This entry allows user peter and all the members of the group operator to gain access to all the program files in the /sbin and /usr/sbin directories, plus the privilege of running the command /usr/local/apps/check.pl. Notice how the trailing slash (/) is required to specify a directory location:

peter, %operator ALL= /sbin/, /usr/sbin, /usr/local/apps/check.pl

Notice also that the lack of any username entries within parentheses () after the = sign prevents the users from running the commands automatically masquerading as another user. This is explained further in the next example.

Granting Access to Specific Files as Another User

The sudo -u entry allows allows you to execute a command as if you were another user, but first you have to be granted this privilege in the sudoers file.

This feature can be convenient for programmers who sometimes need to kill processes related to projects they are working on. For example, programmer peter is on the team developing a financial package that runs a program called monthend as user accounts. From time to time the application fails, requiring "peter" to stop it with the /bin/kill, /usr/bin/kill or /usr/bin/pkill commands but only as user "accounts". The sudoers entry would look like this:

peter ALL=(accounts) /bin/kill, /usr/bin/kill, /usr/bin/pkill

User peter is allowed to stop the monthend process with this command:

[peter@bigboy peter]# sudo -u accounts pkill monthend

Granting Access Without Needing Passwords

This example allows all users in the group operator to execute all the commands in the /sbin directory without the need for entering a password. This has the added advantage of being more convenient to the user:

%operator ALL= NOPASSWD: /sbin/

Using Aliases in the sudoers File

Sometimes you'll need to assign random groupings of users from various departments very similar sets of privileges. The sudoers file allows users to be grouped according to function with the group and then being assigned a nickname or alias which is used throughout the rest of the file. Groupings of commands can also be assigned aliases too.

In the next example, users peter, bob and bunny and all the users in the operator group are made part of the user alias ADMINS. All the command shell programs are then assigned to the command alias SHELLS. Users ADMINS are then denied the option of running any SHELLS commands and su:

Cmnd_Alias    SHELLS = /usr/bin/sh,  /usr/bin/csh, \

/usr/bin/ksh, /usr/local/bin/tcsh, \

/usr/bin/rsh, /usr/local/bin/zsh





User_Alias    ADMINS = peter, bob, bunny, %operator

ADMINS        ALL    = !/usr/bin/su, !SHELLS

This attempts to ensure that users don't permanently su to become root, or enter command shells that bypass sudo's command logging. It doesn't prevent them from copying the files to other locations to be run. The advantage of this is that it helps to create an audit trail, but the restrictions can be enforced only as part of the company's overall security policy.

Other Examples

You can view a comprehensive list of /etc/sudoers file options by issuing the command man sudoers.

Using syslog To Track All sudo Commands

All sudo commands are logged in the log file /var/log/messages which can be very helpful in determining how user error may have contributed to a problem. All the sudo log entries have the word sudo in them, so you can easily get a thread of commands used by using the grep command to selectively filter the output accordingly.

Here is sample output from a user bob failing to enter their correct sudo password when issuing a command, immediately followed by the successful execution of the command /bin/more sudoers.

[root@bigboy tmp]# grep sudo /var/log/messages

Nov 18 22:50:30 bigboy sudo(pam_unix)[26812]: authentication failure; logname=bob uid=0 euid=0 tty=pts/0 ruser= rhost= user=bob

Nov 18 22:51:25 bigboy sudo: bob : TTY=pts/0 ; PWD=/etc ; USER=root ; COMMAND=/bin/more sudoers

[root@bigboy tmp]#

Conclusion

It is important to know how to add users, not just so they can log in to our system. Most server based applications usually run via a dedicated unprivileged user account, for example the MySQL database application runs as user mysql and the Apache Web server application runs as user apache. These accounts aren't always created automatically, especially if the software is installed using TAR files.

Finally, the sudo utility provides a means of dispersing the responsibility of systems management to multiple users. You can even give some groups of users only partial access to privileged commands depending on their roles in the organization. This makes sudo a valuable part of any company's server administration and security policy.

Blog of JoJo 2008-11-19 19:18 发表评论

OpenSSH Public Key Authentication

Blog of JoJo — Wed, 19 Nov 2008 11:17:00 GMT

Secure Shell (SSH) public key authentication can be used by a client to access servers, if properly configured.

Secure Shell (SSH) public key authentication can be used by a client to access servers, if properly configured. These notes describe how to configure OpenSSH for public key authentication, how to enable a ssh-agent to allow for passphrase-free logins, and tips on debugging problems with SSH connections. Password free logins benefit remote access and automation, for example if administering many servers or accessing version control software over SSH.

Public key authenticate can prevent brute force SSH attacks, but only if all password-based authentication methods are disabled. Other options to protect against brute force SSH attacks include pam_tally, or port knocking. Public key authentication does not work well with Kerberos or OpenAFS, which require a password or principal from the client.

Definition of terms used in this documentation:

Client: the system one types directly on, such as a laptop or desktop system.
Server: anything connected to from the client. This includes other servers accessed through the first server connected to.

Never allow root-to-root trust between systems. If required by poorly engineered legacy scripts, limit the from access of the public keys, and if possible only allow specific public keys to run specific commands. Instead, setup named accounts for users or roles, and grant as little root access as possible via sudo.

For more information, see also SSH, The Secure Shell: The Definitive Guide. SSHKeyChain offers integration between the Apple Keychain and OpenSSH.

Public Key Setup

Key Generation | Key Distribution | Key Access Limits

First, confirm that OpenSSH is the SSH software installed on the client system. Key generation may vary under different implementations of SSH. The ssh -V command should print a line beginning with OpenSSH, followed by other details.

$ ssh -V
OpenSSH_3.6.1p1+CAN-2003-0693, SSH protocols 1.5/2.0, OpenSSL 0x0090702f

Key Generation

A RSA key pair must be generated on the client system. The public portion of this key pair will reside on the servers being connected to, while the private portion needs to remain on a secure local area of the client system, by default in ~/.ssh/id_rsa. The key generation can be done with the ssh-keygen(1) utility.

client$ mkdir ~/.ssh
client$ chmod 700 ~/.ssh
client$ ssh-keygen -q -f ~/.ssh/id_rsa -t rsa
Enter passphrase (empty for no passphrase): …
Enter same passphrase again: …

Do not use your account password, nor an empty passphrase. The password should be at least 16 characters long, and not a simple sentence. One choice would be several lines to a song or poem, interspersed with punctuation and other non-letter characters. The ssh-agent setup notes below will reduce the number of times this passphrase will need to be used, so using a long passphrase is encouraged.

The file permissions should be locked down to prevent other users from being able to read the key pair data. OpenSSH may also refuse to support public key authentication if the file permissions are too open. These fixes should be done on all systems involved.

$ chmod go-w ~/
$ chmod 700 ~/.ssh
$ chmod go-rwx ~/.ssh/*

Key Distribution

The public portion of the RSA key pair must be copied to any servers that will be accessed by the client. The public key information to be copied should be located in the ~/.ssh/id_rsa.pub file on the client. Assuming that all of the servers use OpenSSH instead of a different SSH implementation, the public key data must be appended into the ~/.ssh/authorized_keys file on the servers.

# first, upload public key from client to server
client$ scp ~/.ssh/id_rsa.pub server.example.org:

# next, setup the public key on server
server$ mkdir ~/.ssh
server$ chmod 700 ~/.ssh
server$ cat ~/id_rsa.pub >> ~/.ssh/authorized_keys
server$ chmod 600 ~/.ssh/authorized_keys
server$ rm ~/id_rsa.pub

Be sure to append new public key data to the authorized_keys file, as multiple public keys may be in use. Each public key entry must be on a different line.

Many different things can prevent public key authentication from working, so be sure to confirm that public key connections to the server work properly. If the following test fails, consult the debugging notes.

client$ ssh -o PreferredAuthentications=publickey server.example.org
Enter passphrase for key '/…/.ssh/id_rsa': …
…
server$

Key distribution can be automated with module:authkey and CFEngine. This script maps public keys stored in a filesystem repository to specific accounts on various classes of systems, allowing a user key to be replicated to all systems the user has access to.

If exporting the public key to a different group or company, consider removing or changing the optional public key comment field to avoid exposing the default username and hostname.

Key Access Limits

As an optional step to limit usage of the public key for access to any servers, a from statement can be used before public key entries in the ~/.ssh/authorized_keys file on the servers to limit where the client system is permitted to access the server from. Without a from limit, any client system with the appropriate private key data will be able to connect to the server from anywhere. If the keypair should only work when the client system is connecting from a host under example.org, set from="*.example.org" before the public key data.

server$ cat ~/.ssh/authorized_keys
from="*.example.org" ssh-rsa AAAAB3NzaC1…

If a text editor is used to add the from option, ensure the data is saved as a single line; some editors may wrap the public key and thus corrupt the data. Each public key in the ~/.ssh/authorized_keys file must not span multiple lines.

Multiple hosts or addresses can be specified as comma separated values. For more information on the syntax of the from option, see the sshd(8) documentation.

from="*.example.org,10.*,external.example.com" …

Configure `ssh-agent` Process

To reduce the frequency with which the key passphrase must be typed in, setup a ssh-agent(1) daemon to hold the private portion of the RSA key pair for the duration of a session. There are several ways to run and manage ssh-agent, for example from a X11 login script or with a utility like Keychain. These notes rely on the setup of ssh-agent via an @reboot crontab(5) entry, along with appropriate shell configuration.

The ssh-agent must only be run on the client system. The private key of the RSA key pair must remain on the client system. Agent forwarding should be used to make the key available to subsequent logins to other servers from the first server connected to.

Startup cron job

The following crontab(5) entry should run the agent at system startup time. The crond daemon on BSD and Linux systems should support the special @reboot syntax required for this to work.

@reboot ssh-agent -s | grep -v echo > $HOME/.ssh-agent

To setup the agent for the first time without having to reboot the system, run the following.

$ nohup ssh-agent -s > ~/.ssh-agent

Once the ssh-agent is running, any shells already running will need to source in the environment settings from the ~/.ssh-agent file. The SSH_AUTH_SOCK and SSH_AGENT_PID environment variables set in this file are required for the OpenSSH commands such as ssh and ssh-add to communicate with the ssh-agent on the client system.

$ . ~/.ssh-agent

Notes on configuring all shells to be able to run arbitrary commands are available. This reduces the initial setup to the following commands, which can be done from the script reagent.

$ nohup ssh-agent -s | grep -v echo > ~/.ssh-agent
$ allsh - < ~/.ssh-agent

If csh or tcsh is being used instead of a Bourne-based shell, replace the -s argument with -c, and the source command used instead of . in any running shells.

Shell startup script changes

The shell’s startup script on the client system will need to be modified to pull in the required environment settings from ~/.ssh-agent and setup useful aliases. The agent settings in ~/.ssh-agent should not be read in if the client system is being connected to as a server. Remote connections set the SSH_CLIENT environment variable, so ~/.ssh-agent must not be read in when this variable contains data.

[ -z "$SSH_CLIENT" ] && . $HOME/.ssh-agent

alias keyon="ssh-add -t 10800"
alias keyoff='ssh-add -D'
alias keylist='ssh-add -l'

The -t option to ssh-add will remove keys from memory after the specified number of seconds. This option prevents the keys from being left unlocked for long periods of time. Older versions of OpenSSH will not have the timeout -t option.

For the csh and tcsh shells, slightly different configuration of the agent and aliases is required. Consult the relevant ssh-agent(1) and shell documentation.

Once the ssh-agent is running and shell configured to read in the appropriate settings and set easy aliases, enable the key then test a login to a remote server. The keyon will only need to be run when initially adding the private key data to ssh-agent, and only rerun if ssh-agent is restarted or the key is removed with keyoff.

client$ keyon
…
client$ ssh server.example.org
server$ exit
client$ keyoff

Use the keylist command to see what keys are in the agent process.

$ keylist
1024 01:a1:aa:34:21:bc:7d:a4:ea:56:a4:a1:1a:c5:fa:9f /home/…/.ssh/id_rsa (RSA)

If password free logins do not work, see tips on debugging problems with SSH connections to work out where the problem may be.

To make other applications not run from a shell aware of the agent, the environment definitions in the ~/.ssh-agent file will need to be read into the software in question. Consult the documentation for the software to see whether this is possible.

Agent Forwarding

For simple client to server connections, SSH agent forwarding will not be a concern. However, if from the server connected to, one logs into other servers, SSH agent forwarding will need to be enabled. If SSH agent forwarding is disabled, a private key must be available on the proxy system that is recognized by the server being connected to.

To enable forwarding, either use the -A option to ssh when connecting, or set ForwardAgent in an OpenSSH config file, such as ~/.ssh/config. Note that command line arguments override the user-specific configuration file, which in turn can override the global ssh_config configuration file, if any.

Host *
ForwardAgent yes
ForwardX11 no

Agent (and X11) forwarding may represent a security risk, providing more options to an attacker on a compromised server to work back to the client system. If paranoid, disable Agent and X11 forwarding by default, and only enable the features where needed. Also enable StrictHostKeyChecking and use configuration management software such as CFEngine to distribute a global ssh_known_hosts file to all client systems.

Blog of JoJo 2008-11-19 19:17 发表评论

不用密碼直接用 ssh 登入到遠端電腦

Blog of JoJo — Wed, 19 Nov 2008 11:16:00 GMT

參考文件

pinfo ssh-keygen
Linux 伺服器安全防護 (O'REILLY, ISBN: 986-7794-18-4)

多年前 telnet 當道，但在安全意識漸漸浮上檯面之後，telnet 在登入時的安全就被大家質疑，後來 ssh (Secure Shell) 出現時，改變了當初的習慣，大家在管理 Linux 時，現在都使用 ssh 來登入，而 ssh 好處我在這也不再多做說明，因為他還可以配合 rsync 做出遠端備份，一旦設定好 ssh 之後，還會有 scp 可以使用！這樣就可以在不同電腦間 copy 檔案，並且為傳輸的資料加密了！

Shell Script & ssh
自動化的工作可以讓管理員有效率的完成目標，也不用浪費人力和時間做同樣的事情，在無人職守的情況下，要讓 script 自動連入遠端系統做事是件有些麻煩的事，因為您必需登入系統才可以繼續工作，為了不略過登入系統這個步驟，我們可以製做一個 public key 讓遠端的機器信任我們，如此就只要直接連入就可以，而不用再輸入帳號和密碼。

製作 public keys & private keys
利用 ssh-keygen 來做出公用和私有鑰匙，並傳送 public key 到遠端機器使其信任本機登入。

[steven@cute steven]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/steven/.ssh/id_dsa):
Enter passphrase (empty for no passphrase): <- 不用輸入
Enter same passphrase again: <- 不用輸入
Your identification has been saved in /home/steven/.ssh/id_dsa.
Your public key has been saved in /home/steven/.ssh/id_dsa.pub.
The key fingerprint is:
fa:c9:a9:e4:d5:70:52:88:cc:f3:25:fd:68:ae:c4:4b steven@cute.com.tw
[steven@cute steven]$

接著，再到 /home/steven/.ssh 裡看看，會多出 id_dsa 和 id_dsa.pub 這兩個檔案。

[steven@cute steven]$ cd .ssh
[steven@cute .ssh]$ ls
id_dsa id_dsa.pub known_hosts
[steven@cute .ssh]$

現在我們要使遠端機器 mirror.abc.com，使用 sandy 登入時不用輸入密碼，因為，我們應該複製一份 id_dsa.pub 到 sandy@mirror.abc.com 去，並加入到 authorized_keys。

[steven@cute .ssh]$ scp id_dsa.pub sandy@mirror.abc.com:~/id_dsa_steven.pub
sandy@mirror.abc.com's password:
id_dsa.pub 100% |*****************************| 607 00:00
[steven@cute .ssh]$

登入 sandy@mirror.abc.com

[steven@cute .ssh]$ ssh sandy@mirror.abc.com
sandy@mirror.abc.com's password:
-bash-2.05b$ ls id_dsa_steven.pub
id_dsa_steven.pub
-bash-2.05b$ cat id_dsa_steven.pub >> .ssh/authorized_keys
-bash-2.05b$ exit

完成後離開，回到本機，再做一次 ssh 到 mirror.abc.com

[steven@cute .ssh]$ ssh sandy@mirror.abc.com
-bash-2.05b$

如此就不用輸入密碼就直接登入了！

保護你的私有金匙
在製做 dsa key 時，會有一份私有和一份公有金匙，實務上會保留起來，並做備份，因為當 ssh 在登入時，會使用 id_dsa.pub 和本機的 id_dsa 做確認，因此如果這兩者比對不成功時就會再次要求輸入密碼。

Blog of JoJo 2008-11-19 19:16 发表评论

双系统怎样卸载linux

Blog of JoJo — Sun, 02 Nov 2008 14:57:00 GMT

Q: 装了xp（C盘）和fedora4双系统，用grub进行引导，现在想要删掉fedora回收硬盘空间，并且也不要grub引导就能直接进xp，不知道应该怎么操作呢？

A1: 删除Linux 分区，然后xp 光盘启动，进修复控制台运行 fixmbr 命令。

A2: win下右键单击我的电脑——>管理——>磁盘管理。这样可以看见硬盘的分区情况，把属于linux的分区删除。合并成win下的分区。这样硬盘空间就回收了
至于不用grub引导的问题，找一张xp系统盘进入dos下用 fdisk /mbr命令就可以删除grub。

Blog of JoJo 2008-11-02 22:57 发表评论

BlogJava-jojo's blog－－快乐忧伤都与你同在-文章分类-Linux 技术相关

每日一记 2009/07/09 Sed Command Tutorial/Example

SED Tutorial

每日一记 2009/06/16 Linux的时间设置与同步 (NTP)

每日一记 2009/06/06 linux服务器配置之Sendmail配置

Rsync命令参数详解

每日一记 2009/06/03 screen处理后台任务

遍历两个日期之间天数的算法

Memcache工作原理

1 Memcache是什么

2 Memcache工作原理

3 如何使用

建立Manager类

建立数据对象

Mencache Introduction

What is memcached?

How it Works

Porting the Application

Shouldn't the database do this?

What about shared memory?

What about MySQL 4.x query caching?

What about database replication?

Is memcached fast?

What about race conditions?

how to monitor ibm mq from nagios

One Response to “how to monitor ibm mq from nagios”

用nagios来监控网络服务器和网络服务

linux下开机自动启动Oracle脚本

Preventing access to .svn folders or a file in Apache

Allow/Deny Directive in

Pair programming and agile

User Story Estimation Techniques

Increase file descriptor limit under Linux to prevent java.net.SocketException: Too many open files Increase file descriptor limit under Linux to prevent java.net.SocketException: Too many open files Increase file descriptor limit under Linux to prev

Increase file descriptor limit under Linux to prevent

java.net.SocketException: Too many open files

Get current limit:

Get current number of open file descriptors:

Increase the limit:

crontab

svn中的branch和tag

linux 开机 自动 启动apache 脚本

RRDTool

在MySql上实现Replication(Master 与 Slave 数据同步)

正则表达式详解（Perl）

Synchronizing Large Applications

Synchronizing Large Applications

Using the doNotRemoveList Flag

How to Automate Secure File Synchronization using SSH and rsync

How to Automate Secure File Synchronization using SSH and rsync

Introduction

Create a Key

Using rsync With SSH and Your Key

Additional Information

Acknowledgements

Appendix A: File restrict-to-rsync

Remote Backups With Rsync

Remote Backups With Rsync

PHP Requirements

PHP Requirements

[转载]Ubuntu/Debian包管理命令大全（apt & dpkg）

Linux_Users_and_Sudo

Introduction

Who Is the Super User?

How To Add Users

How to Change Passwords

How to Delete Users

How to Tell the Groups to Which a User Belongs

How to Change the Ownership of a File

Using sudo

Simple Sudo Examples

Temporarily Gaining root Privileges

Becoming root for a Complete Login Session

Downloading and Installing the sudo Package

The /etc/sudoers File

Table 9-1 Format of the /etc/sudoers File

Simple /etc/sudoers Examples

Granting All Access to Specific Users

Granting Access To Specific Users To Specific Files

Granting Access to Specific Files as Another User

Granting Access Without Needing Passwords

What is `memcached`?

Is `memcached` fast?

linux 开机自动启动apache 脚本

Appendix A: File `restrict-to-rsync`