BlogJava-Skynet-随笔分类-perl

shell txt 分析小结

刘凯毅 — Thu, 26 Nov 2009 03:27:00 GMT

刘凯毅 2009-11-26 11:27 发表评论

perl 使用小结

刘凯毅 — Wed, 11 Nov 2009 05:36:00 GMT

环境设置：如果有系统字符编码冲突，在当前
vim ~/.bash_profile 下加入
LANG=zh_CN
LC_ALL=zh_CN.UTF8
export LANG LC_ALL

字符编码转化:
  # 由decode解析,默认会使用系统编码输出
  # 在 linux 下面其实等价 encode("UTF-8", decode("GBK",$_));
perl -MEncode -ne 'print decode("GBK",$_);'  file.txt

判断数据是否符合输出：
  echo "121" |perl -ne 'print if /2/;'   #print 123

匹配正则group输出:
  echo "abc121a" |perl -ne 'print $1 if /(\D+)/;'   #print abc

大小写转化：
  # 全部大小转小写
  echo "ABC1C2cGJ" |perl -ne 'tr/[A-Z]/[a-z]/; print ;'  #print
  # "L 中间全部小写 "E   ; "U 中间全部大写 "E    ↓
  echo "ABC1C2cGJ" |perl -ne 's/(.*?1)(.*?)(2.*?)/$1\L$2\E$3/g; print ;'  #print ABC1c2cGJ

源文件替换：
   echo "ABC 123" > te
   sed -i 's/ABC/abc/g' te
       或者 : perl -i -pe 's/ABC/abc/g' te
   cat te # print abc 123

外部传参:
tt="cc"
echo "gg" |perl -ne ' print "'$tt'" ;'
输出：cc

  perl -e  'print "$ARGV[0]\t$ARGV[1]\n" '  'par1' 'par2'  #print par1 par2

重复列输出:
   cat xx.txt | awk -F"    " 'a[$1]++'
   或者 :
   cat xx.txt |perl -F"\t" -ane  '$a{$F[1]}++;END{
        while(($k,$v)=each(%a)){ print "$k = $v "n"; }
    }'
    结果比如：
       百度手机在线 = 7
       中兴 = 2
       万信恒通  = 2
   还比如：查看各用户有多少个进程
   ps -ef |perl  -ane  '$a{$F[0]}++;END{
      while(($k,$v)=each(%a)){ print "$k = $v \n"; }
   }'

求两项  交集
cat BuyMusic.20090525| perl -ne 'BEGIN{
$p1="600902000005416300";
$p2="600902000006211983";
$p_col=30;
$mob_col=0;

}END{
  my @inter = grep {$a{$_}} keys %b; # 求交集
  #print $p1,"=",join(",",keys %a),""n";
  #print $p2,"=",join(",",keys %b),""n";
  print "产品 $p1:",scalar keys %a," "n";
  print "产品 $p2:",scalar keys %b," "n";
  print "交集:",scalar @inter," "n";
}
chomp;
@lis=split /\|<>\|/ ;
if( $lis[ $p_col] eq $p1 ){
   $a{$lis[$mob_col]}++;
}
if( $lis[$p_col] eq $p2 ){
   $b{$lis[$mob_col]}++;
}
'

关键字 Top 10 ，输出源文本数据 :
perl -e  '
  my $num=10; # top 10
  open(MYFILE, "<$ARGV[0]");
  open(MYFILE2, "<$ARGV[0]");

  # 关键字列数
  while(){@lis=split /\|<>\|/;$fck{$lis[1]}++ }

  foreach $k (sort { $fck{$b} <=> $fck{$a} } keys %fck){
     if(++$row>$num){last; }
     $arr[@arr] = $k;
  }

  while(){@lis=split /\|<>\|/;
     if(grep { $arr[$_] eq $lis[1] } 0..$#arr){
       # print "$fck{$lis[1]}:$_"; #带关键字出现次数输出
       print ;
     }
  }
' qdSearch.log

刘凯毅 2009-11-11 13:36 发表评论

hadoop streaming( hadoop + perl )小试

刘凯毅 — Fri, 25 Sep 2009 06:33:00 GMT

参考:
http://hadoop.apache.org/common/docs/r0.15.2/streaming.html

注意
目前 streaming 对 linux pipe #也就是 cat |wc -l 这样的管道不支持，但不妨碍我们使用perl,python 行式命令！！
原话是：
Can I use UNIX pipes? For example, will -mapper "cut -f1 | sed s/foo/bar/g" work?
    Currently this does not work and gives an "java.io.IOException: Broken pipe" error.
    This is probably a bug that needs to be investigated.
但如果你是强烈的 linux shell pipe 发烧友！参考下面
$> perl -e 'open( my $fh, "grep -v null tt |sed -n 1,5p |");while ( <$fh> ) {print;} '
     #不过我没测试通过！！

环境：hadoop-0.18.3
$> find . -type f -name "*streaming*.jar"
./contrib/streaming/hadoop-0.18.3-streaming.jar

测试数据：

-bash-3.00$ head tt
null    false    3702    208100
6005100    false    70    13220
6005127    false    24    4640
6005160    false    25    4820
6005161    false    20    3620
6005164    false    14    1280
6005165    false    37    7080
6005168    false    104    20140
6005169    false    35    6680
6005240    false    169    32140
......

运行：

c1=" perl -ne  'if(/.*\t(.*)/){\$sum+=\$1;}END{print \"\$sum\";}' "
# 注意这里 $ 要写成 \$    " 写成 \"
echo $c1; # 打印输出 perl -ne 'if(/.*"t(.*)/){$sum+=$1;}END{print $sum;}'
hadoop jar hadoop-0.18.3-streaming.jar
   -input file:///data/hadoop/lky/jar/tt
   -mapper   "/bin/cat"
   -reducer "$c1"
   -output file:///tmp/lky/streamingx8

结果:
cat /tmp/lky/streamingx8/*
1166480

本地运行输出:
perl -ne 'if(/.*"t(.*)/){$sum+=$1;}END{print $sum;}' < tt
1166480

结果正确!!!!

命令自带文档：

-bash-3.00$ hadoop jar hadoop-0.18.3-streaming.jar -info
09/09/25 14:50:12 ERROR streaming.StreamJob: Missing required option -input
Usage: $HADOOP_HOME/bin/hadoop [--config dir] jar \
          $HADOOP_HOME/hadoop-streaming.jar [options]
Options:
  -input    <path>     DFS input file(s) for the Map step
  -output   <path>     DFS output directory for the Reduce step
  -mapper   <cmd|JavaClassName>      The streaming command to run
  -combiner <JavaClassName> Combiner has to be a Java class
  -reducer  <cmd|JavaClassName>      The streaming command to run
  -file     <file>     File/dir to be shipped in the Job jar file
  -dfs    <h:p>|local  Optional. Override DFS configuration
  -jt     <h:p>|local  Optional. Override JobTracker configuration
  -additionalconfspec specfile  Optional.
  -inputformat TextInputFormat(default)|SequenceFileAsTextInputFormat|JavaClassName Optional.
  -outputformat TextOutputFormat(default)|JavaClassName  Optional.
  -partitioner JavaClassName  Optional.
  -numReduceTasks <num>  Optional.
  -inputreader <spec>  Optional.
  -jobconf  <n>=<v>    Optional. Add or override a JobConf property
  -cmdenv   <n>=<v>    Optional. Pass env.var to streaming commands
  -mapdebug <path>  Optional. To run this script when a map task fails
  -reducedebug <path>  Optional. To run this script when a reduce task fails
  -cacheFile fileNameURI
  -cacheArchive fileNameURI
  -verbose

刘凯毅 2009-09-25 14:33 发表评论

部分高级正则特性使用

刘凯毅 — Thu, 27 Aug 2009 08:04:00 GMT

一些特殊正则元字符说明:
1. *? 和 +? 和 {n,}?  懒惰匹配
   1.1 非懒惰　　　　　　　　　　　　　　　   ↓
    echo "ab2c121a" |perl -ne 'print $1 if /(.*)"d/;'   #print ab2c12
   1.2 懒惰　　　　　　　　　　　　　　　　      ↓
    echo "ab2c121a" |perl -ne 'print $1 if /(.*?)"d/;'   #print ab
2.  回溯引用和前后查找:
   2.1 向前查找   (?=..)    　　　　　　　　　　　　   ↓
    echo "ab2c121a" |perl -ne 'print $1 if /(.*?)(?=2)/;'  #print ab
   2.2 向后查找 (?<=..)  　　     　　　　　　↓
    echo "ab2c121a" |perl -ne 'print $1 if /(?<=2)(.*)(?=2)/;' #print c1
   2.3 负向-前/后查找  (?!..) (?)
    #不能匹配 ..                               ↓
    echo "ab2c121a" |perl -ne 'print $1 if /(?'    #print 无
    echo "ab2c121a" |perl -ne 'print $1 if /(?'    #print c121a
   2.4 条件 ?()  = if   ?()| = if else
    # ?()  例如

必须同时出现 ↓ ↓
echo "

"|perl -ne 'print $2 if /(

)?("w*)(?(1)<"/p>)/' #print xx
echo "

xx"|perl -ne 'print $2,""n" if /(

)?("w*)(?(1)<"/p>)/' #print 空
echo "xx"|perl -ne 'print $2 if /(

)?("w*)(?(1)<"/p>)/'    #print xx
    # ?()|  例如还是上面的，
    # 当有

可以接

也可以接数字结尾 ↓
echo "

xx1

"|perl -ne 'print $2 if /(

)?("w*)(?(1)<"/p>|"d)/' #print xx1
echo "

xx1"|perl -ne 'print $2 if /(

)?("w*)(?(1)<"/p>|"d)/' # print xx

刘凯毅 2009-08-27 16:04 发表评论

perl vs php (转)

刘凯毅 — Fri, 15 May 2009 15:09:00 GMT

Perl/Php Translation

arrays

hashes

data structures

array split/join

case conversion

string comparisons

functions

string matching operations

basename/dirname

environment variables

POST/GET parameters

HTML elements

URL encode

MySQL database access

— Perl arrays —	— Php arrays —
@a = (); @a = ( 'xx', 11, 33.5, ); @a = 12..33; $a[2] = 'something'; $len = scalar(@a); # or $len = @a; @a3 = ('xx', @a1, @a2); ($x, $y) = @a; $a[@a] = 'new'; # push push pop shift unshift splice foreach $i (@a) { .. }	$a = array(); $a = array( 'xx', 11, 33.5, ); $a = range(12,33); $a[2] = 'something'; $len = count($a); $a3 = array_merge('xx', $a1, $a2); list($x, $y) = $a; $a[] = 'new'; # push array_push array_pop array_shift array_unshift array_splice foreach ($a as $i) { .. }

— Perl arrays —

— Php arrays —

@a = ();

            

            @a = ( 'xx', 11, 33.5, );

            

            @a = 12..33;

            

            $a[2] = 'something';

            

            $len = scalar(@a);

            # or

            $len = @a;

            

            @a3 = ('xx', @a1, @a2);

            

            ($x, $y) = @a;

            

            $a[@a] = 'new'; # push

            

            push

            pop

            shift

            unshift

            splice

            

            foreach $i (@a) { .. }

$a = array();

            

            $a = array( 'xx', 11, 33.5, );

            

            $a = range(12,33);

            

            $a[2] = 'something';

            

            $len = count($a);

            

            

            

            $a3 = array_merge('xx', $a1, $a2);

            

            list($x, $y) = $a;

            

            $a[] = 'new'; # push

            

            array_push

            array_pop

            array_shift

            array_unshift

            array_splice

            

            foreach ($a as $i) { .. }

— Perl hashes —	— Php hashes —
%h = (); %h = ( 'x' => 'y', 'z' => 'w', ); $h{'x'} = 7; while (($key,$value) = each(%h)) { .. } $a = keys(%h); $b = values(%h); delete $h{'x'};	$h = array(); $h = array( 'x' => 'y', 'z' => 'w', ); $h['x'] = 7; foreach ($h as $key => $value) { .. } $a = array_keys($h); $b = array_values($h); unset( $h['x'] );

— Perl hashes —

— Php hashes —

%h = ();

            

            %h = ( 'x' => 'y',

            'z' => 'w',

            );

            

            $h{'x'} = 7;

            

            while (($key,$value) = each(%h))

            { .. }

            

            $a = keys(%h);

            $b = values(%h);

            

            delete $h{'x'};

$h = array();

            

            $h = array( 'x' => 'y',

            'z' => 'w',

            );

            

            $h['x'] = 7;

            

            foreach ($h as $key => $value)

            { .. }

            

            $a = array_keys($h);

            $b = array_values($h);

            

            unset( $h['x'] );

— Perl data structures —	— Php data structures —
%h = ('a'=>13, 'b'=>25); @x = ('hi', 'there', 'all',); @mix = ( %h, @x, [33..39], { x=>15, yy=>23, }, ); $mix[0]->{'b'} # == 25 $mix[0]{'b'} # == 25 $mix[2]->[2] # == 35 $mix[2][2] # == 35	$h = array('a'=>13, 'b'=>25); $x = array('hi', 'there', 'all',); $mix = array($h, $x, range(33,39), array('x'=>15, 'yy'=>23), ); $mix[0]['b'] # == 25 $mix[2][2] # == 35

— Perl data structures —

— Php data structures —

%h = ('a'=>13, 'b'=>25);

            @x = ('hi', 'there', 'all',);

            

            @mix = ( %h, @x,

            [33..39],

            { x=>15, yy=>23, },

            );

            

            $mix[0]->{'b'}  # == 25

            $mix[0]{'b'}    # == 25

            $mix[2]->[2]    # == 35

            $mix[2][2]      # == 35

$h = array('a'=>13, 'b'=>25);

            $x = array('hi', 'there', 'all',);

            

            $mix = array($h, $x,

            range(33,39),

            array('x'=>15, 'yy'=>23),

            );

            

            $mix[0]['b']  # == 25

            

            $mix[2][2]    # == 35

— Perl array split/join —	— Php array split/join —
@a = split( '\|', $s ); @a = split( 's+', $s ); $s = join( '\|', @a );	$a = preg_split( '/\|/', $s, -1, PREG_SPLIT_NO_EMPTY ); $a = preg_split( '/s+/', $s, -1, PREG_SPLIT_NO_EMPTY ); $s = join( '\|', $a );

— Perl array split/join —

— Php array split/join —

@a = split( '|', $s );

            

            @a = split( 's+', $s );

            

            

            $s = join( '|', @a );

$a = preg_split( '/|/', $s,

            -1, PREG_SPLIT_NO_EMPTY );

            $a = preg_split( '/s+/', $s,

            -1, PREG_SPLIT_NO_EMPTY );

            

            $s = join( '|', $a );

— Perl case conversion —	— Php case conversion —
$s = lc($s); $s = uc($s); $s =~ tr/a-z/A-Z/;	$s = strtolower($s); $s = strtoupper($s);

— Perl string comparisons —	— Php string comparisons —
$s1 eq $s2 $s1 lt $s2	strcmp($s1,$s2) == 0 # or $s1 === $s2 strcmp($s1,$s2) < 0

— Perl functions —	— Php functions —
sub foo { my @args = @_; } sub foo { $x = 5; } foo2( @a, %h );	function foo() { $args = func_get_args(); } function foo() { global $x; $x = 5; } function foo2($x, $y) { } foo2( $a, $h );

— Perl string matching operations —	— Php string matching operations —
$s =~ m/(w+)/; $substr = $1; @all = ($s =~ m/(w+)/g); $s =~ s/s+/X/; $s =~ s/s+/X/g; $s =~ s/^s+\|s+$//g;	preg_match( "/(w+)/", $s, $match ); $substr = $match[1]; preg_match_all( "/(w+)/", $s, $match ); $all = $match[0]; $s = preg_replace( "/s+/", 'X', $s, 1 ); $s = preg_replace( "/s+/", 'X', $s ); $s = trim($s);

— Perl string matching operations —

— Php string matching operations —

$s =~ m/(w+)/;

            $substr = $1;

            

            @all = ($s =~ m/(w+)/g);

            

            

            $s =~ s/s+/X/;

            $s =~ s/s+/X/g;

            

            $s =~ s/^s+|s+$//g;

preg_match( "/(w+)/", $s, $match );

            $substr = $match[1];

            

            preg_match_all( "/(w+)/", $s, $match );

            $all = $match[0];

            

            $s = preg_replace( "/s+/", 'X', $s, 1 );

            $s = preg_replace( "/s+/", 'X', $s );

            

            $s = trim($s);

— Perl basename/dirname —	— Php basename/dirname —
use File::Basename; $b = basename($path); $d = dirname($path);	$b = basename($path); $d = dirname($path);

— Perl environment variables —	— Php environment variables —
%ENV $ENV{REQUEST_METHOD} $ARGV[$i] $0	$_SERVER $_SERVER[REQUEST_METHOD] $argv[$i+1] $argv[0] # Php/CGI only

— Perl POST/GET parameters —	— Php POST/GET parameters —
#form/hyperlink parameters: # s : single-valued # m : multi-valued use CGI (:standard); $s = param('s'); @m = param('m'); @param_names = param(); $num_params = param();	#form/hyperlink parameters: # s : single-valued # m[] : multi-valued # (such as multi-selections # and checkbox groups) $PARAM = array_merge($_GET, $_POST); $s = $PARAM['s']; # a scalar $m = $PARAM['m']; # an array $param_names = array_keys($PARAM); $num_params = count($PARAM);

— Perl POST/GET parameters —

— Php POST/GET parameters —

#form/hyperlink parameters:

            # s : single-valued

            # m : multi-valued

            

            use CGI (:standard);

            

            

            

            

            $s = param('s');

            @m = param('m');

            

            @param_names = param();

            $num_params = param();

#form/hyperlink parameters:

            # s   : single-valued

            # m[] : multi-valued

            #       (such as multi-selections

            #        and checkbox groups)

            

            $PARAM

            = array_merge($_GET, $_POST);

            

            $s = $PARAM['s'];  # a scalar

            $m = $PARAM['m'];  # an array

            

            $param_names = array_keys($PARAM);

            $num_params = count($PARAM);

— Perl HTML elements —

— Php HTML elements —

use CGI (:standard);

            

            

            

            

            

            

            

            $ref = "x.cgi";

            a({href=>$ref}, "yy")

            

            textfield({name=>"yy", size=>5})

            

            password({name=>"yy", size=>5})

            

            textarea({name=>"yy",

            cols=>5, rows=>2})

            

            submit({value=>"yy"})

            

            button( {name=>"xx",

            value=>"yy",

            onclick=>"submit()",

            }

            )

            

            %labels = (0=>'a',1=>'q',2=>'x');

            popup_menu( { name=>"xx",

            values=>[0..2],

            labels=>%labels,

            size=>4,

            }

            )

            

            

            @a = ('xx','yy','zz');

            radio_group( { name=>'nn',

            values=> @a,

            default=>'_',

            linebreak=>1,

            }

            )

            

            %labels = ('xx'=>'L1','yy'=>'L2');

            @a = keys( %labels );

            checkbox_group( { name=>'nn',

            values=> @a,

            labels=> %labels,

            }

            )

            

            table(

            Tr(

            [

            td(['a','b']),

            td(['x','y']),

            ]

            )

            )

# The Perl/CGI functions have the

            # additional property of "stability"

            # when used in reentrant forms.

            # The values of the HTML elements are

            # set according to the incoming

            # parameter values for those elements.

            # The versions below are not stable.

            

            $ref = "x.php";

            yy

            

            

            

            

            

            <br />
            <br />
            

            

            

            

            
            name="xx" value="yy"

            onclick="submit()">

            

            

            

            

            

            $a = array('xx','yy','zz');

            foreach ($a as $_)

            echo "
            name=nn value='$_'>$_
";

            

            

            

            

            $labels = array('xx'=>'L1','yy'=>'L2');

            foreach (array_keys($labels) as $_)

            echo "
            name=nn value='$_'>",

            $labels[$_];

            

            

            

            








            
            a b
            
            
            
            x y

— Perl URL encode —	— Php URL encode —
use URI::Escape; uri_escape($val) uri_unescape($val)	urlencode($val) urldecode($val)

— Perl MySQL database access —	— Php MySQL database access —
use DBI; $dbh = DBI->connect( 'DBI:mysql:test:localhost', $usr,$pwd ); $dbh->do( $sql_op ) $query = $dbh->prepare( $sql_op ); $query->execute(); while( @record = $query->fetchrow() ) { .. } $dbh->quote($val)	$dbh = mysql_connect( 'localhost', $usr, $pwd ); mysql_query('USE test') mysql_query( $sql_op ); $results = mysql_query( $sql_op ); while($record = mysql_fetch_row($results)) { .. } "'" . addslashes($val) . "'"

— Perl MySQL database access —

— Php MySQL database access —

use DBI;

            $dbh = DBI->connect(

            'DBI:mysql:test:localhost',

            $usr,$pwd

            );

            

            $dbh->do( $sql_op )

            

            $query = $dbh->prepare( $sql_op );

            $query->execute();

            

            while(

            @record = $query->fetchrow() )

            { .. }

            

            

            $dbh->quote($val)

 

            $dbh = mysql_connect(

            'localhost', $usr, $pwd

            );

            mysql_query('USE test')

            

            mysql_query( $sql_op );

            

            $results = mysql_query( $sql_op );

            

            

            while($record =

            mysql_fetch_row($results))

            { .. }

            

            

            "'" . addslashes($val) . "'"

刘凯毅 2009-05-15 23:09 发表评论

perl 使用

刘凯毅 — Wed, 01 Apr 2009 06:12:00 GMT

对应脚本运用：
1. shell 统筹管理脚本的运行。合理结合 crontab , ps -ef ,kill 等命令。
2. perl 处理短小快。
3. python 有比较复杂结构和逻辑的。

本文主要介绍 perl 的行级命令使用，力求短小快：

#最简单的
$ perl -e 'print "Hello World\n"'

#处理文件行
$ perl -n -e 'print $_' file1

#编码转换
#如果有需要在使用下 encode("UTF-8", decode("GBK",$_));在 linux 下默认 utf-8
perl -MEncode -ne 'print decode("GBK",$_);' file.txt

#正则使用
# if($_=~/.*\/(.*)$/){ print $1 ;} 这是perl 巨方便的地方 $1 xx
# next LINE 跳到下一个循环
$ perl -n -e 'next LINE unless /pattern/; print $_'

#去除换行 chomp
perl -e 'print split(/\n/,"asg\n");'

#像 awk 一样 Begin End
$ perl -ne 'END { print $t } @w = /(\w+)/g; $t += @w' file.txt

#像 awk -F"x" 一样切割行
#-a 打开自动分离 (split)  模式
#@F 为切割后的数组
perl -F'\t' -ane '
if($F[1]=~/侃侃/ and $F[2]=~/爱情啊/){
      print "$F[3]\t$F[4]\t$F[5]\n"
  }
' all2_data.sort.st

实际处理：

perl -F'\|\|' -ane '
my $actor,$music ;
if  ( $F[3] ){
  $music=$F[2];
  $actor=$F[3];
}else{
  $music=$F[0];
  $actor=$F[1];
}
  $music =~ tr/[A-Z]/[a-z]/;
  $music =~ s/\s*(.*)\s*$.*$/\1/g;
  $actor =~ tr/[A-Z]/[a-z]/;
  $actor =~ s/\s*(.*)\s*$.*$/\1/g;
print "$actor-$music\n";
' ring.utf8.txt  |sort -u  > ring.actor_music.sort.utf8.txt &
wc -l ring.actor_music.sort.utf8.txt

#像 sed 一样替换
# -i 和 sed 一样 perl 输出会替换调源 file.txt
$ perl -i -pe 's/\bPHP\b/Perl/g' file.txt

#外部传参
perl -ne'print "$ARGV[0]\t$ARGV[1]\n" ' file.txt 'par1' 'par2'
#结果 .. par1 par2 ..

# 查询出重复列次数，并列举出来
cut -d"     "  -f 2 .collection_mobile.data |perl -ne '
   END{
     while (($key,$value)=each(%a)){print $key,"=",$value,"\n";};
   }BEGIN{ %a =(); }
   chomp;
   $a{$_}+=1;
'

结果
Ring=532895
CRBT=68500
RingBoxes=880
Song=96765

#一些实际使用 :)
find . -name "*.mp3" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
perl -F'\t' -ane 'if($F[1]=~/$ARGV[0]/ and $F[2]=~/$ARGV[1]/){print "$F[3]\t$F[4]\t$F[5]\n"}' all2_data.sort.st '侃侃' '爱情啊'

#与 find 合用 -e $ARGV[0] 批量把 excel 倒成文本格式
find . -maxdepth 1 -name "*xls" -exec perl -e '

require("/home/xj_liukaiyi/src/perl/excel/excelUtil.pl");
my $file=$ARGV[0];
sub myRead{
  my $str = "";
  for $tmp(@_){
    $str="$str$tmp\t";
  }
  $str="$str\n";
  system "echo \"$str\" >> $file.data ";
}
&parse_excel("$file",0,\&myRead);
print "$file\n";

' {} \;

参考：
http://www.ibm.com/developerworks/cn/linux/sdk/perl/l-p101/index.html
http://bbs.chinaunix.net/viewthread.php?tid=499434

刘凯毅 2009-04-01 14:12 发表评论

perl 一些有用的 util

刘凯毅 — Tue, 17 Mar 2009 05:49:00 GMT

比如日志，和shell运行

#!/bin/perl -w
use Time::Local;

my $log_input_file="~/log";
sub set_log_input_file{
  $log_input_file=$_[0];
}

#&perl_log("tt","/home/xj_liukaiyi/log");
sub perl_log{
  my $log_msg=shift @_;
  print $log_msg,"\n";
  system 'ttmp1=$(date +%Y-%k-%d);ttmp2=$(date +%M:%S);echo "$ttmp1 $ttmp2:'."$log_msg".'" >>'.$log_input_file;
}

#&system_util("xls -l","~/log");
sub system_util{
  my $comm=shift @_;
  my @return=system $comm;
  if ( $return[(@return-1)] == 0 ) {
     &perl_log("[success]".$comm,$log_input_file);
  }else{
     &perl_log("[error]".$comm,$log_input_file);
  }
  return $return[(@return-1)];
}

1;

excel 读取util
require("/home/xj_liukaiyi/src/perl/excel/excelUtil.pl");&parse_excel("2w.xls","歌曲列表","&read_wlh);

#!/bin/perl -w
use Spreadsheet::ParseExcel;
use Spreadsheet::ParseExcel::FmtUnicode;
use Encode;
use Unicode::Map();
my $Map = new Unicode::Map("GB2312");

my $excelCODE='gb2312';
my $coslCODE='utf8';

my %parseCache ;
my %writeCache ;

my $oFmtJ = Spreadsheet::ParseExcel::FmtUnicode->new(Unicode_Map =>$excelCODE);
my $oExcel = new Spreadsheet::ParseExcel;

#
#&parse_excel("all4.xls","乐扑","&read_lp);
# sub read_lp{ $_[1 .. 2.3 ] }
sub parse_excel{
my $sExcelName=shift;
my $sSheetName=shift;
my $action_fun=shift;

my($iR, $iC, $oWkS,$oWkC);
#开启 excel
my $oBook = $oExcel->Parse($sExcelName,$oFmtJ);
#根据 sheet 名称匹配
for(my $iSheet=0; $iSheet < $oBook->{SheetCount} ; $iSheet++){
    my $t_oWkS = $oBook->{Worksheet}[$iSheet];
    my $sWkSName=encode($coslCODE,decode($excelCODE,$t_oWkS->{Name} ));
    if($sWkSName=~/$sSheetName/){
      $oWkS=$t_oWkS;
      last;
   }
}

for(my $iR = $oWkS->{MinRow} ;
     defined $oWkS->{MaxRow} && $iR <= $oWkS->{MaxRow} ;
     $iR++){
     my @arr_row;
     for(my $iC = $oWkS->{MinCol} ;
         defined $oWkS->{MaxCol} && $iC <= $oWkS->{MaxCol} ;
         $iC++){
       $oWkC=$oWkS->{Cells}[$iR][$iC];
       my $col_value="";
           $col_value=encode($coslCODE,decode($excelCODE,$oWkC->Value) ) if ($oWkC) ;
           push(@arr_row,$col_value);
         }
      push(@arr_row,$iR);
     $action_fun->(@arr_row);
    }
}

#使用 demo
find . -maxdepth 1 -name "*xls" -exec perl -e '
require("/home/xj_liukaiyi/src/perl/excel/excelUtil.pl");
my $file=$ARGV[0];
sub myRead{
  my $str="";
  foreach$tmp(@_){
    $str+"$tmp\t";
  }
  system "echo $str >> $file.data";
}
&parse_excel("$file",0,\&myRead);
print "$file\n";
' {} \;

刘凯毅 2009-03-17 13:49 发表评论

perl 工具小脚本

刘凯毅 — Mon, 02 Mar 2009 09:51:00 GMT

工具小脚本

#!/bin/perl
use File::Find;
use Log::FileSimple;

#use strict;
#use warnings;

# 说明:
#     遍历出 "/cygdrive/c/Documents and Settings/Administrator/"
#       满足 "^.*\.txt$" 添加文件
#         并运行 "ls -l $fileAllName "
#    运行日志 ./log
#
# perl this.pl
#     "ls -l $fileAllName "
#     "/cygdrive/c/Documents and Settings/Administrator/"
#     "^.*\.txt$"
#     ./log
#
# run_comm 参数1 替换说明：
#     $filePath  为输入参数2，遍历根目录 "/cygdrive/c/Documents and Settings/Administrator/"
#    $fileAllName 为当前遍历，满足正则条件的全路径+文件名
#    $logicFileAllName  为当前遍历，满足正则条件的逻辑路径+文件名
#    $logicFilePath 为当前遍历，逻辑路径
#    $logicFileName 文件名称
#    $logicFileExp  文件后缀
#        （test.txt 文件
#            $logicFileName 为 test
#            $logicFileExp 为 txt
#            ）

my $run_comm=$ARGV[0];
my $file_path=$ARGV[1];
my $file_find_rex=$ARGV[2];
my $logfile=$ARGV[3];

my $log = new Log::FileSimple(name=> 'Log::FileSimple logs',
                              file      => "$logfile",
                              mask      => -1,
                              autoflush => 0,
                              );

sub fileThumb{
eval{
  $_tmp_run_comm=$run_comm ;
  if( /$file_find_rex/ ){
    $fileNum++;
    $_tmp_run_comm=~s/\$filePath/$file_path/g;

    my $_file_all_name=$File::Find::name;
    $_tmp_run_comm=~s/\$fileAllName/$_file_all_name/g;
    my $_logic_file_all_name=$_file_all_name;
          $_logic_file_all_name=~s/$file_path//g;
        $_logic_file_all_name=~s/^\s*\.//g;
    $_tmp_run_comm=~s/\$logicFileAllName/\.\/$_logic_file_all_name/g;
    if($_logic_file_all_name=~/^(.*)\/([^\/]*)\.([^\.]*)$/){
      my $_logic_file_path=$1;
      my $_logic_file_name=$2;
      my $_logic_file_exp=$3;
      $_tmp_run_comm=~s/\$logicFilePath/\.\/$_logic_file_path/g;
      $_tmp_run_comm=~s/\$logicFileName/$_logic_file_name/g;
      $_tmp_run_comm=~s/\$logicFileExp/$_logic_file_exp/g;
      }
    #print $_tmp_run_comm,"\n";
    my @return=eval{system("$_tmp_run_comm 2>&1 \|echo \$\? ")};
    if ( $return[(@return-1)] != 0 ) {
      $log->log(message=>"success($fileNum):$_file_all_name");
       }else{
      $log->log(message=>"error($fileNum):$_file_all_name");
        }
  }
}
}

find(\&fileThumb,$file_path);

刘凯毅 2009-03-02 17:51 发表评论

soap (java,perl,要写代码还不过 100 char)

刘凯毅 — Sun, 01 Mar 2009 16:00:00 GMT

可能最简单的 soap 哦！
服务器为 java tomcat
下载服务器：http://www.apache.org/dist/ws/
   axis/ 为java
   axis-c/    为 c

1.我这下载的 axis 1_4 版本，解压 /webapps/axis 放到 tomcat 的webapps
启动tomcat（这就不修改任何，如有需要该 web.xml 等配置了）参考 >>
    blog 主还有很多介绍 axis 大家可以看看日志 > 其下的 "学习"

直接写个文件 HelloWorld.jws 放到 %TOMCAT_HOME%"webapps"axis

public class HelloWorld {
     public String sayHello()   {
     return "HELLO soap!";
}
}

等不及可以使用看看
http://127.0.0.1:8080/axis/HelloWorld.jws?method=sayHello
结果：

<soapenv:Envelope>
<soapenv:Body>
<sayHelloResponse soapenv:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<sayHelloReturn xsi:type="xsd:string">HELLO soap!sayHelloReturn>
sayHelloResponse>
soapenv:Body>
soapenv:Envelope>

客户段
perl cpan >> install SOAP::Lite

#!perl -w
use SOAP::Lite;
print SOAP::Lite
    -> proxy('http://127.0.0.1:8080/axis/HelloWorld.jws')
    -> uri('http://127.0.0.1:8080/axis/HelloWorld.jws')
    -> sayHello()
    -> result;

结果： HELLO soap！

刘凯毅 2009-03-02 00:00 发表评论

Memcached 对话 Google ProtocolBuffers (perl)

刘凯毅 — Thu, 26 Feb 2009 05:45:00 GMT

看看我这张 web 系统架构图，那就知道使用统一存储格式的好处了。
在这种结构中你就单纯使用 java ，或 php 某一种单一语言描述格式，存储你集群交互的数据吗！？使用ProtocolBuffers 吧，Google 已经有现成的api来帮你扩展了。

参考：http://blog.csdn.net/lcj8/archive/2009/02/17/3900157.aspx

作为 Memcached 就是为集群使用，那么作为在缓存存储的数据我想也应该适用与各种环境，这时候数据结构协议 Google ProtocolBuffers
参考：http://code.google.com/apis/protocolbuffers/docs/overview.html
其他语言：http://code.google.com/p/protobuf/wiki/OtherLanguages
（可适用与多种语言，
javascript 读取缓存中的数据 ,你可以想象下你的 web服务开发的将来！）

目前学习perl 中就先上个这的代码：
参考：
http://search.cpan.org/~gariev/Google-ProtocolBuffers-0.08/lib/Google/ProtocolBuffers.pm#___top

#!/bin/perl -w
use Cache::Memcached;
use Google::ProtocolBuffers;
#开启 memd
my $memd = new Cache::Memcached{servers => ['10.0.2.15:11211'] };

#声明缓存存储 protocol 格式
my $key='test';
Google::ProtocolBuffers->parse("
  message Person{
    required int32 id           =1;
    required string name        =2;
  }
",{create_accessors=>1});

my $data=Person->encode({
  id=>123,
  name=>'liukaiyi'
});

#缓存存储
$memd->add($key =>$data,3600);

#######################################################################

my $memd = new Cache::Memcached{servers => ['10.0.2.15:11211'] };
my $person;{
#从缓存中取出
  $person = Person->decode($memd->get($key));
}

print $person->id,"-",$person->name;

结果：
123-liukaiyi

刘凯毅 2009-02-26 13:45 发表评论

memcached 使用（perl）

刘凯毅 — Wed, 25 Feb 2009 08:23:00 GMT

这大体上可以看出，服务器端口缓存技术。

memcached 官方：http://www.danga.com/memcached/
安装前，先安装 libevent
其上为 linux 软件一般安装，看他们readme文档
$>memcached -d -u nobody -m 512 127.0.0.1 -p 11211

如果有异常，
ln -s /usr/local/lib/libevent-1.4.so.2 /lib/libevent-1.4.so.2
参考 http://blog.chinaunix.net/u2/70049/showart_1665279.html
我这就使用 perl 语言了，
其他语言参考 http://code.google.com/p/memcached/wiki/Clients

perl 使用 cpan> install Cache::Memcached ;#会使用 perl 我就不说了
代码说明：不停对 key 为test 的值进行递增

#!/bin/perl -w
use Cache::Memcached;

my $memd = new Cache::Memcached{servers => ['127.0.0.1:11211'] };
my $key = 'test';
$memd->add($key => 1,3600) or warn 'Alread added';
while(1){
print $memd->get($key),"\n";
$memd->incr($key) or warn 'FAIL!';
}

使用 telnet 127.0.0.1 11211
   $> get test
   VALUE test 0 2
   97 #这就是我们递增的值,当然你使用什么语言取都是一样的
   END

刘凯毅 2009-02-25 16:23 发表评论

perl file find

刘凯毅 — Mon, 23 Feb 2009 15:55:00 GMT

使用 File::Find

use File::Find;
use Time::Piece;

$timeOneDay = 86400;
$dayMultipler = 1;

$ByteCount = 0;
$fileAll = 0;
$fileJpg = 0;
$fileYes = 0;
$pathToTra = ".";

$gt = localtime;
$t1 = $gt->epoch;
find(\&fileThumb, $pathToTra);
$gt = localtime;
$t2 = $gt->epoch;

print "Total KByte: ".($ByteCount/1024)."\n";
print "Total Scan:  ".$fileAll."\n";
print "Total Jpeg:  ".$fileJpg."\n";
print "Total Match: ".$fileYes."\n";
print "Total TimeCost: ".($t2-$t1)."\n";

sub fileThumb
{
   $fileAll++;
   if ((/.*\.jpg$/) )
   {
      $fileJpg++;
      my $lt = 0;
      $lt = localtime;

      my ($dev, $ino, $mode, $nlink, $uid, $gid, $rdev, $size, $atime, $mtime, $ctime, $blksize, $blocks) = stat($_)  or die "Unable to stat $_\n";
      if ( $mtime < ($lt->epoch - $timeOneDay*$dayMultipler) )
      {
         print "[ $fileYes ]"."$File::Find::name";
         print $size." ".$ctime." ".$lt->epoch."\n";
         $ByteCount += $size;
         $fileYes++;
      }
   }
}

刘凯毅 2009-02-23 23:55 发表评论

log4j perl 错误日志提取

刘凯毅 — Thu, 19 Feb 2009 05:01:00 GMT

日志

#!/bin/perl
use Encode ;
open(NOTESFILES,$ARGV[0]);

my %errorMeganotes;

print ">>>>>>>>>>>>>>",$ARGV[0],"<<<<<<<<<<\n";

while (<NOTESFILES>) {
  if( $_=~/^(\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2})\s+ERROR\s+(.*)\s/g ){
     if($errorMeganotes{$2}){
       push @{$errorMeganotes{$2}},$1;
     }else{
       $errorMeganotes{$2}=[$1];
     }
  }
}

for(keys %errorMeganotes){
  my $tmp_keys=$_;
  my @tmp_array= @{$errorMeganotes{$tmp_keys}};
  print "error number :>>",scalar(@tmp_array),"<<\n",encode("GBK",decode("UTF-8",$tmp_keys)),"\n",join(",\t",@tmp_array),"\n\n\n";
}

#!/bin/sh
nowDate=$(date -d '-1 day' +%Y-%m-%d);
echo $nowDate

perl /data/apps/shell/meganotesErrorLog.pl /data/apps/meganotes/tomcat/logs/app/app.log.$nowDate >  _tmp_$nowDate ;
perl /data/apps/shell/meganotesErrorLog.pl /data/apps/schedule/tomcat/logs/app/app.log.$nowDate  >> _tmp_$nowDate ;

mail -s $nowDate-ERROR-log  my@mail.com  <  _tmp_$nowDate;
rm _tmp_$nowDate ;

结果：
>>>>>>>/data/apps/meganotes/tomcat/logs/app/app.log.2009-02-18<<<<<<<<<
error number :>>2<<
MusicActionOpLogProxy:142 - music.popularTime=[Invalid field value for field "music.popularTime".]
2009-02-18 15:53:54, 2009-02-18 16:00:12
。。。。。。

刘凯毅 2009-02-19 13:01 发表评论

使用 awk 一样使用 perl 并她可爱的正则

刘凯毅 — Tue, 17 Feb 2009 09:39:00 GMT

参考：
http://my.huhoo.net/archives/2007/08/perl_1.html

简单的
   echo "tt" |perl -ne 'print $_ '
多行
   find . -type f |perl -ne 'print $1 $_ if $_=~ /(.*)flv/'
限定行数
   find . -type f |perl -ne 'print $_ if $_=~ /.*flv/ && $tmp++<10 && $tmp>4'
    #当然shell>> find . -type f |grep ".*flv" |sed -n '5,10p' 和上面是一样的
    # 不过你看那其中使用 $1 等正则重定向 ,哈哈
结果大体上
./test3/0.flv
./flvplayer.swf
./test.flv
./0.flv

perl -h
-e program        one line of program (several -e's allowed, omit programfile)
-n                assume "while (<>) { ... }" loop around program
-p                assume loop like -n but print line also, like sed

刘凯毅 2009-02-17 17:39 发表评论

perl log4j 分析日志

刘凯毅 — Tue, 17 Feb 2009 07:50:00 GMT

普通的 log4j 日志；
当然特殊要求特殊处理。

#!/bin/perl
open(FILES,"/data/apps/schedule/tomcat/logs/app/app.log.2009-02-17");
while (<FILES>) {
    next if !/^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\s+ERROR\s+/;
    if($_=~/(.*-){3}(.*?):(.*)$/g){
        $error{$2}++ ;
    }elsif($_=~/(.*-){3}(.*)/g ){
        $error{$2}++ ;
    }
    。。。。。
    }
print "$_: $error{$_}\n" for(keys %error);

java.net.SocketTimeoutException: 4
test1 illegal!: 4
cd illegal!: 4
test2 illegal!: 4
MusicAlbum not exists: 5
Not Found Any File: 14
java.net.ConnectException: 3
Album not exists: 2
。。。

刘凯毅 2009-02-17 15:50 发表评论