500 Error on Confluence Startup (KR)

기존에 AWS EC2의 m3.large에 jira, confluence를 설치형으로 운영하고 있었다.

(nginx, mysql도 같은 인스턴스에 설치하여 운용)

 

최근들어서 메모리 부족 이슈가 자꾸 발생한다.

•  java.lang.OutOfMemoryError: Java heap space

일단은 스케줄 작업 시간을 분산해보았다.

confluence schedule job

 

몇일 괜찮더니 여전히 또 메모리 부족으로 confluence 접근이 안된다.

EC2 사양을 높이고 m3.large -> m3.xlarge로 (7.5G -> 15G) 메모리를 높였다.

 

메모리 사용하는 프로세스

• java/confluence
• java/jira
• nginx
• mysql

 

startup.sh 명령을 실행해서 띄웠다.

 

안되고 500 ERROR가 난다.

29-Jul-2019 01:51:06.145 SEVERE [http-nio-8090-exec-2] org.apache.catalina.core.StandardHostValve.custom Exception Processing ErrorPage[errorCode=500, location=/500page.jsp]  com.atlassian.util.concurrent.LazyReference$InitializationException: java.lang.NullPointerException         at com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:149)         at com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:112)         at com.atlassian.confluence.plugin.servlet.filter.ServletFilterModuleContainerFilter.getServletModuleManager(ServletFilterModuleContainerFilter.java:23)         at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:68)         at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)         at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:50)         at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)         at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:721)         at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:468)         at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:391)         at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:318)         at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:439)         at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:305)         at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:399)         at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)         at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)         at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)         at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)         at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)         at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1521)         at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1478)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)         at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException         at com.atlassian.spring.container.ContainerManager.getComponent(ContainerManager.java:33)         at com.atlassian.confluence.util.LazyComponentReference$Accessor.get(LazyComponentReference.java:46)         at com.atlassian.util.concurrent.Lazy$Strong.create(Lazy.java:85)         at com.atlassian.util.concurrent.LazyReference$Sync.run(LazyReference.java:321)         at com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:143)         ... 29 more

 

아...... 일단 구글링

https://www.google.com/search?newwindow=1&ei=pFg-XcOuMpXMvwTrhKbABA&q=org.apache.catalina.core.StandardHostValve.custom+Exception+Processing+ErrorPage%5BerrorCode%3D500%2C+location%3D%2F500page.jsp%5D&oq=org.apache.catalina.core.StandardHostValve.custom+Exception+Processing+ErrorPage%5BerrorCode%3D500%2C+location%3D%2F500page.jsp%5D&gs_l=psy-ab.3...12614.12614..12886...0.0..0.0.0.......0....2j1..gws-wiz.Xc4qbAH0AaA&ved=0ahUKEwjDkOSgidnjAhUV5o8KHWuCCUgQ4dUDCAo&uact=5

org.apache.catalina.core.StandardHostValve.custom Exception Processing ErrorPage[errorCode=500, location=/500page.jsp] - Google

 

다행히 1번째 결과가 나의 상황과 유사하다.

링크를 열어보니 희망이 생긴다.

https://community.atlassian.com/t5/Confluence-questions/500-Error-on-Confluence-Startup/qaq-p/393445

500 Error on Confluence Startup

 

링크의 댓글을 따라가본다. (That did it for me, thanks! 라고 한다. 이거 왠지 될거 같다.)

• https://confluence.atlassian.com/confkb/how-to-clear-confluence-plugins-cache-297664846.html?_ga=2.182736742.1356167257.1564365849-966154368.1561701121

How to clear Confluence plugins cache - Atlassian Documentation

왠지 느낌이 이거같다. 플러그인 캐쉬 문제

 

solution for this error

다 풀릴것 같은 문제가 <confluence-home>을 못찾고 또 지연된다.

결국 find로 뒤져보니 저 디렉토리가 똑같이 존재한다.

 

언급된 디렉토리를 삭제하진 않고 일단 mv 명령어로 다른 path로 옮기고

confluence를 재시작하였다.

 

시작하면서 저 디렉토리들을 새로 만든다.

그러다보니 기동하는데 오래걸린다.  (참고로 default 8090 port로 띄움)

 

띄우고나서 확인, 아까는 500 ERROR였는데 200 OK 뜬다. 

$ wget http://localhost:8090 --2019-07-29 04:19:04--  http://localhost:8090/ Resolving localhost (localhost)... 127.0.0.1 Connecting to localhost (localhost)|127.0.0.1|:8090... connected. HTTP request sent, awaiting response... 302 Found Location: http://localhost:8090/login.action?os_destination=%2Findex.action&permissionViolation=true [following] --2019-07-29 04:19:04--  http://localhost:8090/login.action?os_destination=%2Findex.action&permissionViolation=true Reusing existing connection to localhost:8090. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] Saving to: ‘index.html’ index.html                                     [ <=>                                                                                    ]  19.64K  --.-KB/s   in 0.002s  2019-07-29 04:19:04 (11.1 MB/s) - ‘index.html’ saved [20107]

 

잘된다.