paulwong

2020年2月16日 #

!!21 MOST IMPORTANT JAVA 8 VM OPTIONS FOR SERVERS

27/03/2016 MAURICE KNOPP 7 KOMMENTARE

In this video I explain some 21 JVM parameters which are suited for most server applications. If you have any questions, you can read those links below for more information or just ask in the comments section.

I run several Java enterprise server applications. I often wondered – what are the best „default“ JVM settings for a server application to start with in production? I read a lot on the web and tried several things myself and wanted to share what I found out, so far. Links containing more information about JVM optimization can be found here:

http://blog.sokolenko.me/2014/11/javavm-options-production.html

http://www.petefreitag.com/articles/gctuning/

http://stas-blogspot.blogspot.de/2011/07/most-complete-list-of-xx-options-for.html

So let’s start:

-server

Use „-server“: All 64-bit JVMs use the server VM as default anyway. This setting generally optimizes the JVM for long running server applications instead of startup time. The JVM will collect more data about the Java byte code during program execution and generate the most efficient machine code via JIT.

-Xms=<heap size>[g|m|k] -Xmx=<heap size>[g|m|k]

The „-Xmx/-Xms“ settings specify the maximum and minimum values for the JVM heap memory. For servers, both params should have the same value to avoid heap resizing during runtime. I’ve applications running with 16GB heap sizes without an issue.

Depending on your application, you will have to try out how much memory will be best suited for your use case.

-XX:MaxMetaspaceSize=<metaspace size>[g|m|k]

Java 8 has no „Permanent Generation“ (PermGen) anymore but requires additional „Metaspace“ memory instead. This memory is used, in addition to the heap memory we specified before, for storing class meta data information.

The default size will be unlimited – I tend to limit MaxMetaspaceSize with a somewhat high value. Just in case something goes wrong with the application, the JVM will not hog all the memory of the server.

I suggest: Let your application run for a couple of days to get a feeling for how much Metaspace Size it uses normally. Upon next restart of the application set the limit to e.g. double the value.

-XX:+CMSClassUnloadingEnabled

Additionally, you might want to allow the JVM to unload classes which are held in memory but no code is pointing to them any more. If your application generates lots of dynamic classes, this is what you want.

-XX:+UseConcMarkSweepGC

This option makes the JVM use the ConcurrentMarkSweepGC – It can do much work in parallel to program execution but in some circumstances a „full GC“ with a „STW pause“ might still occur. I’ve read many articles and came to the conclusion that this GC is still the best one for server workloads.

-XX:+CMSParallelRemarkEnabled

The option CMSParallelRemarkEnabled means the remarking is done in parallel to program execution – which is what you want if your server has many cores (and most servers do).

 -XX:+UseCMSInitiatingOccupancyOnly  -XX:CMSInitiatingOccupancyFraction=<percent>

Normally the GC will use heuristics to know when it’s time to clear memory. GC might kick in too late with default settings (causing full-Gcs).
Some sources say it might be a good idea to disable heuristics altogether and just use generation occupancy to start a CMS collection cycle. Setting values around 70% worked fine for all of my applications and use cases.

-XX:+ScavengeBeforeFullGC

The first option tells the GC to first free memory by clearing out the „young generation“ or fairly new objects before doing a full GC.

-XX:+CMSScavengeBeforeRemark

CMSScavengeBeforeRemark does attempt a minor collection before the CMS remark phase – thus keeping the remark pause afterwards short.

-XX:+CMSClassUnloadingEnabled

The option „-XX:+CMSClassUnloadingEnabled“ here tells the JVM to unload classes, which are not needed any more by the running application. If you deploy war files to an application server like wildfly, tomcat or glassfish without restarting the server after the deployment, this flag is for you.

-XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses

The option „-XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses“ is especially important if your application uses RMI (remote method invocation). The usage of RMI will cause the JVM to do a FULL-GC EVERY HOUR! This might be a very bad idea for large heap sizes because the FULL-GC pause might take up to several seconds. It would be better to do a concurrent GC and try to unload unused classes to free up more memory – which is exactly what the second option does.

-XX:+PrintGCDateStamps -verbose:gc -XX:+PrintGCDetails -Xloggc:"<path to log>"

These options shown here will write out all GC related information to a specified log file. You can see how well your GC configuration works by looking into it.

I personally prefer to use the „Visual GC“ plug in for the „Visual VM“ tool to monitor the general JVM and GC behavior.

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<path to dump>`date`.hprof

When your JVM runs out of memory, you will want to know why. Since the OOM error might be hard to reproduce and you want to get your production server up and running again – you should specify a path for a heap dump. When things have settled down, you can analyze the dump afterwards.

-Djava.rmi.server.hostname=<external IP> -Dcom.sun.management.jmxremote.port=<port>

These options will help you to specify an IP and port for JMX – you will need those ports open to connect remotely to a JVM running on a server for tools like VisualVM. You can gain deep insights over cpu and memory usage, gc behaviour, class loading, thread count and usage of your application this way.

Visual VM
Lastly, I would like to recommend to you the VisualVM tool which is bundled with the Java 8 JDK. You can use it to gain more insights about your specific application behaviour on the JVM – like cpu and memory usage, thread utilisation and much more.

VisualVM can be extended with a plug in called „Visual GC“. It will briefly show you VERY detailed information about the usage of the young and old generation object spaces. You can easily spot problems with garbage collection simply by analyzing these graphs during application runtime.

Thank you very much for watching! If you liked the video you might consider giving it a „thumbs up“. If you have any questions – just put them in the comments section. I will reply as quickly as possible.

-------------------------------------------------------

-XX:+UseCompressedOops [If Max Heap allocation is less than 32GB]
This can save a significant amount of memory and this option should already be enabled by default on recent java 8 versions. This option allowes object references to be stored as 32-bit values instead of 64-bit on 64-bit JVMs. This leads to before mentioned memory savings.

-XX:+AggressiveOpts
This option will enable performance options which are hoped to become enabled by default in upcoming released of the JVM. This option sets some performance settings but is marked as experimental! So you should only enable it, when you have to possibility to test your application thoroughly before enabling this flag on an production server.

-XX:+UseStringDeduplication
Since Java 8 update 20 you can use this option to reduce the memory usage of your application. The JVM will spot identical strings in memory, remove the duplicated and point all references to the remaining, single instance of the string.

-XX:+UseG1GC
Will tell the JVM to use the most recent G1 garbage collector. You are trading better application response times (due to shorter gc times with G1) against lower throughput (compared against good old ConcMarkSweepGC / CMS). If your application can deliver more value through short gc times, then G1 is definately better suited. Otherwise on Java 8, I’d recommend sticking with CMS.

Concerning your Tomcat 8 question, I’d suggest you have a look into it with the „VisualVM“ tool. Look at memory usage, GC times (visual GC plugin), pull and analyse stack traces or thread dumps to find the weak spot. You might also consider attaching a debugger to tomcat to find the bug.

https://www.maknesium.de/21-most-important-java-8-vm-options-for-servers

posted @ 2020-02-16 22:30 paulwong 阅读(17) | 评论 (0) | 编辑收藏

2020年2月14日 #

windows下jenkins提示文件名太长

由于jenkins是调用windows的git取代码，因此是git的问题，进行如下配置即可：

git config --global core.longpaths true

posted @ 2020-02-14 14:37 paulwong 阅读(17) | 评论 (0) | 编辑收藏

2020年2月7日 #

SPRING BOOT 环境下减少中间件依赖的UNIT测试

SPRING BOOT 环境下，测试有时会依赖于外部的中间件，如Mysql，Activemq，Mongodb等，那如何能减少这种依赖呢？
SPRING BOOT其实已经实现了自动化配置。

Mongodb

SPRING BOOT的自动化配置文件：org.springframework.boot.autoconfigure.mongo.embeddedEmbedded.MongoAutoConfiguration.java

在pom.xml中新增一test profile，并添加相应jar包，这样可防止对其他profile的影响，如果是在Eclipse跑测试，需在Project的属性中指定Active Profile为test，以覆盖pom.xml的定义。
这种方式即使是使用SPRING DATA MONGODB的REPOSITORY也是适用的。

  <profile>
        <id>test</id>
        <dependencies>
            <dependency>
                <groupId>de.flapdoodle.embed</groupId>
                <artifactId>de.flapdoodle.embed.mongo</artifactId>
                <scope>test</scope>
            </dependency>
        </dependencies>
        <activation>
            <activeByDefault>false</activeByDefault>
        </activation>
    </profile>

在application-test.yaml中添加端口，其他如IP那些信息都不需要

spring:
   data:
      mongodb:
         port: 27017

unit test config

import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.Reader;
import java.io.UncheckedIOException;
import java.nio.charset.StandardCharsets;
import java.util.List;

import javax.annotation.PostConstruct;
import javax.sql.DataSource;

import org.bson.Document;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
import org.springframework.core.io.Resource;
import org.springframework.core.io.ResourceLoader;
import org.springframework.data.mongodb.core.MongoTemplate;
import org.springframework.data.mongodb.repository.config.EnableMongoRepositories;
import org.springframework.jdbc.datasource.init.DatabasePopulatorUtils;
import org.springframework.jdbc.datasource.init.ResourceDatabasePopulator;
import org.springframework.util.FileCopyUtils;

@Configuration
@Profile({"test", "integrationTest"})
@EnableMongoRepositories(
        basePackages = {"paul.com.repository"
        }
)
public class EmbeddedDataSourceConfiguration {

    @Value("classpath:/initdata/USER.json")
    private Resource userResource;

    @Value("classpath:/initdata/MEMBERS.json")
    private Resource membersResource;

    @Autowired
    private ResourceLoader resourceLoader;

    @Autowired
    private DataSource dataSource;

    @Autowired
    private MongoTemplate  mongoTemplate;

    @PostConstruct
    protected void initialize() throws FileNotFoundException, IOException {
        this.initializeHsqldb();
        this.initializeMongodb();
    }

    private void initializeHsqldb() {
        ResourceDatabasePopulator populator = new ResourceDatabasePopulator();
        populator.addScript(resourceLoader.getResource("classpath:/org/springframework/batch/core/schema-hsqldb.sql"));
        populator.setContinueOnError(true);
        DatabasePopulatorUtils.execute(populator , dataSource);
    }

    private void initializeMongodb() throws FileNotFoundException, IOException {
        this.saveResource(userResource, "USER");

        this.saveDocumentList(membersResource, "MEMBER");
    }

    private void saveResource(Resource resource, String collectionName) {
        String resourceJson = this.asString(resource);
        Document resourceDocument = Document.parse(resourceJson);
        this.mongoTemplate.save(resourceDocument, collectionName);
    }

    private void saveDocumentList(Resource resource, String collectionName) {
        String resourceJson = this.asString(resource);
        Document resourceDocument = Document.parse("{ \"list\":" + resourceJson + "}");
        List<Document> documentList = resourceDocument.get("list", List.class);
        documentList.forEach(document -> this.mongoTemplate.save(document, collectionName));
    }

    private String asString(Resource resource) {
        try (Reader reader = new InputStreamReader(resource.getInputStream(), StandardCharsets.UTF_8)) {
            return FileCopyUtils.copyToString(reader);
        } catch (IOException e) {
            throw new UncheckedIOException(e);
        }
    }

//    @Bean(destroyMethod="close")
//    public DataSource dataSource() {
//        BasicDataSource dataSource = new BasicDataSource();
//        dataSource.setDriverClassName(environment.getProperty("batch.jdbc.driver"));
//        dataSource.setUrl(environment.getProperty("batch.jdbc.url"));
//        dataSource.setUsername(environment.getProperty("batch.jdbc.user"));
//        dataSource.setPassword(environment.getProperty("batch.jdbc.password"));
//        return dataSource;
//    }
}

ActiveMQ

只需更改application-test.yml中的brokerUrl为vm://embedded即可

spring:
   activemq:
      broker-url: vm://embedded?broker.persistent=false,useShutdownHook=false
      in-memory: true
      non-blocking-redelivery: true
      #packages:
        #trust-all: false
        #trusted: com.memorynotfound
      pool:
        block-if-full: true
        block-if-full-timeout: -1
        create-connection-on-startup: true
        enabled: false
        expiry-timeout: 0
        idle-timeout: 30000
        max-connections: 1
        maximum-active-session-per-connection: 500
        reconnect-on-exception: true
        time-between-expiration-check: -1
        use-anonymous-producers: true
        user: admin
        #password: ENC(hWJHuMyhydTqyF32neasTw==)
        password: admin

关系型数据库

将在application-test.yml中的数据库信息删除，同时在pom.xml中添加jar包依赖，这边是采用HSQL数据库

  <profile>
        <id>test</id>
        <dependencies>
            <dependency>
                <groupId>org.hsqldb</groupId>
                <artifactId>hsqldb</artifactId>
                <scope>test</scope>
            </dependency>
        </dependencies>
        <activation>
            <activeByDefault>false</activeByDefault>
        </activation>
    </profile>

非SPRING BOOT/SPRING的纯JDK环境可参考
https://github.com/yandex-qatools/embedded-services

https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo

https://github.com/jonyfs/spring-boot-data-embedded-mongodb/blob/master/src/main/java/br/com/jonyfs/spring/boot/data/embedded/mongodb/config/MongoConfig.java

ActiveMQ:
https://memorynotfound.com/spring-boot-embedded-activemq-configuration-example/

posted @ 2020-02-07 10:28 paulwong 阅读(114) | 评论 (0) | 编辑收藏

2020年2月3日 #

配置SPRING BATCH中的JUNIT TEST

关键是JobLauncherTestUtils的配置：

@Configuration
    public class BatchTestConfiguration {


        @Bean
        public JobLauncherTestUtils stoppedReportJobLauncherTestUtils(
                JobLauncher stoppedReportJobLauncher
        ) {
            return new JobLauncherTestUtils() {

                @Autowired
                public void setJobLauncher(JobLauncher stoppedReportJobLauncher) {
                    super.setJobLauncher(stoppedReportJobLauncher);
                }

                @Autowired
                public void setJob(Job stoppedReportJob) {
                    super.setJob(stoppedReportJob);
                }

            };
        }
    }

posted @ 2020-02-03 16:47 paulwong 阅读(19) | 评论 (0) | 编辑收藏

2020年1月21日 #

Transform RemoteChunk to remote with json format in Spring Batch

Spring Batch Remote Chunk模式下，远程执行JOB时，传输的对象是ChunkRequest/ChunkResponse，无法转成JSON格式传输。需要自定义Transformer：

JsonToChunkRequestTransformer.java

package com.frandorado.springbatchawsintegrationslave.transformer;

import java.io.IOException;
import java.util.Collection;
import java.util.Map;
import java.util.stream.IntStream;

import org.springframework.batch.core.ExitStatus;
import org.springframework.batch.core.StepContribution;
import org.springframework.batch.core.StepExecution;
import org.springframework.batch.integration.chunk.ChunkRequest;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.integration.aws.support.AwsHeaders;
import org.springframework.integration.json.JsonToObjectTransformer;
import org.springframework.messaging.Message;
import org.springframework.stereotype.Component;

import com.amazonaws.services.sqs.AmazonSQSAsync;
import com.fasterxml.jackson.databind.ObjectMapper;

@Component
public class JsonToChunkRequestTransformer extends JsonToObjectTransformer {

    private static final String MESSAGE_GROUP_ID_HEADER = "message-group-id";

    @Autowired
    AmazonSQSAsync amazonSQSAsync;

    @Override
    protected Object doTransform(Message<?> message) throws Exception {
        // ACK
        ack(message);

        return this.getMessageBuilderFactory().withPayload(buildChunkRequest(message)).setHeader(MESSAGE_GROUP_ID_HEADER, "unique").build();
    }

    private ChunkRequest buildChunkRequest(Message<?> message) throws IOException {
        Map map = new ObjectMapper().readValue(message.getPayload().toString(), Map.class);
        Map stepContributionMap = (Map) map.get("stepContribution");
        Map exitStatusMap = (Map) stepContributionMap.get("exitStatus");

        StepContribution stepContribution = new StepContribution(new StepExecution("null", null));
        ExitStatus exitStatus = new ExitStatus((String) exitStatusMap.get("exitCode"), (String) exitStatusMap.get("exitDescription"));

        IntStream.range(0, (Integer) stepContributionMap.get("readCount")).forEach(e -> stepContribution.incrementReadCount());
        stepContribution.incrementWriteCount((Integer) stepContributionMap.get("writeCount"));
        stepContribution.incrementFilterCount((Integer) stepContributionMap.get("filterCount"));
        stepContribution.incrementReadSkipCount((Integer) stepContributionMap.get("readSkipCount"));
        IntStream.range(0, (Integer) stepContributionMap.get("writeSkipCount")).forEach(e -> stepContribution.incrementWriteSkipCount());
        IntStream.range(0, (Integer) stepContributionMap.get("processSkipCount"))
                .forEach(e -> stepContribution.incrementProcessSkipCount());
        stepContribution.setExitStatus(exitStatus);

        return new ChunkRequest((Integer) map.get("sequence"), (Collection) map.get("items"), (Integer) map.get("jobId"), stepContribution);
    }

    private void ack(Message<?> message) {
        String receiptHandle = message.getHeaders().get(AwsHeaders.RECEIPT_HANDLE, String.class);
        String queue = message.getHeaders().get(AwsHeaders.QUEUE, String.class);
        String queueUrl = amazonSQSAsync.getQueueUrl(queue).getQueueUrl();

        amazonSQSAsync.deleteMessage(queueUrl, receiptHandle);
    }
}

JsonToChunkResponseTransformer.java

package com.frandorado.springbatchawsintegrationmaster.transformer;

import java.io.IOException;
import java.util.Map;

import org.springframework.batch.core.StepContribution;
import org.springframework.batch.core.StepExecution;
import org.springframework.batch.integration.chunk.ChunkResponse;
import org.springframework.integration.json.JsonToObjectTransformer;
import org.springframework.messaging.Message;
import org.springframework.stereotype.Component;

import com.fasterxml.jackson.databind.ObjectMapper;

@Component
public class JsonToChunkResponseTransformer extends JsonToObjectTransformer {

    @Override
    protected Object doTransform(Message<?> message) throws Exception {
        return buildChunkResponse(message);
    }

    private ChunkResponse buildChunkResponse(Message<?> message) throws IOException {
        Map map = new ObjectMapper().readValue(message.getPayload().toString(), Map.class);

        Integer jobId = (Integer) map.get("jobId");
        Integer sequence = (Integer) map.get("sequence");
        String messageContent = (String) map.get("message");
        Boolean status = (Boolean) map.get("successful");

        StepContribution stepContribution = new StepContribution(new StepExecution("-", null));

        return new ChunkResponse(status, sequence, Long.valueOf(jobId), stepContribution, messageContent);
    }
}

https://frandorado.github.io/spring/2019/07/29/spring-batch-aws-series-introduction.html
https://github.com/frandorado/spring-projects/tree/master/spring-batch-aws-integration/spring-batch-aws-integration-master/src/main/java/com/frandorado/springbatchawsintegrationmaster/transformer

posted @ 2020-01-21 16:44 paulwong 阅读(138) | 评论 (0) | 编辑收藏

如何加CRON JOB到LINUX

https://linuxtools-rst.readthedocs.io/zh_CN/latest/tool/crontab.html

posted @ 2020-01-21 15:50 paulwong 阅读(31) | 评论 (0) | 编辑收藏

2020年1月15日 #

Linux and Unix du command tutorial with examples

https://shapeshed.com/unix-du/

查询>100M的文件，倒序排列，文件大小以K/M/G等显示：
-h: 以可读方式显示，-t 排除小于100m的文件，sort排序，less: 分页

du -ah -t 100m / | sort -n -r | less

posted @ 2020-01-15 17:17 paulwong 阅读(41) | 评论 (0) | 编辑收藏

2020年1月10日 #

SPRING INTEGRATION ERROR HANDLING

https://github.com/zakyalvan/spring-integration-java-dsl-learn

package com.jwebs.learn.errorhandling;

import java.util.Random;

import javax.jms.ConnectionFactory;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.integration.annotation.IntegrationComponentScan;
import org.springframework.integration.channel.PublishSubscribeChannel;
import org.springframework.integration.core.MessageSource;
import org.springframework.integration.dsl.IntegrationFlow;
import org.springframework.integration.dsl.IntegrationFlows;
import org.springframework.integration.dsl.core.Pollers;
import org.springframework.integration.dsl.jms.Jms;
import org.springframework.integration.support.MessageBuilder;
import org.springframework.messaging.MessagingException;

/**
* Show how to handle error in spring integration flow.
* Please note, errorChannel in spring integration only applicable to
* error thrown in asynch component.
*
* @author zakyalvan
*/
@SpringBootApplication
@IntegrationComponentScan
public class ErrorHandlingApplication {
    public static void main(String[] args) throws Exception {
        ConfigurableApplicationContext applicationContext = new SpringApplicationBuilder(ErrorHandlingApplication.class)
                .web(false)
                .run(args);

        Runtime.getRuntime().addShutdownHook(new Thread(() -> applicationContext.close()));

        System.out.println("Pres enter key to exit

");
        System.in.read();
        System.exit(0);
    }

    @Autowired
    private ConnectionFactory connectionFactory;

    @Bean
    public MessageSource<Integer> randomIntegerMessageSource() {
        return () -> MessageBuilder.withPayload(new Random().nextInt()).build();
    }

    @Bean
    public IntegrationFlow withErrorFlow() {
        return IntegrationFlows.from(randomIntegerMessageSource(), spec -> spec.poller(Pollers.fixedDelay(1000)))
                    .handle(Jms.outboundGateway(connectionFactory)
                    .requestDestination("processor.input")
                    .replyContainer(spec -> spec.sessionTransacted(true)))
                    .get();
    }

    @Autowired
    @Qualifier("errorChannel")
    private PublishSubscribeChannel errorChannel;

    @Bean
    public IntegrationFlow errorHandlingFlow() {
        return IntegrationFlows.from(errorChannel)
                .handle(message -> System.out.println("@@@@@@@@@@@@@@@@@@@@@" + ((MessagingException) message.getPayload()).getFailedMessage().getPayload()))
                .get();
    }
}

posted @ 2020-01-10 15:32 paulwong 阅读(36) | 评论 (0) | 编辑收藏

2019年12月30日 #