Skip to content

fix: deploy.sh를 2프로세스 롤링 재시작 방식으로 변경 - #201

Merged
unam98 merged 1 commit into
mainfrom
fix/rolling-restart-deploy
Jul 28, 2026
Merged

fix: deploy.sh를 2프로세스 롤링 재시작 방식으로 변경#201
unam98 merged 1 commit into
mainfrom
fix/rolling-restart-deploy

Conversation

@unam98

@unam98 unam98 commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

배경

prod가 단일 인스턴스+단일 프로세스 구조에서, nginx + 2프로세스(8080/8081) 구조로 무중단 전환됨 (DNS 이중등록 기반 블루-그린 전환, 새 인스턴스 runnect-prod로 태그 교체 완료).

기존 scripts/deploy.sh는 실행 중인 jar 프로세스를 전부 찾아 한 번에 종료하고 새 프로세스 1개만 재기동하는 방식이라, 2프로세스 구조에 그대로 쓰면:

  • 배포 시 두 프로세스가 동시에 죽어 nginx 뒤에 살아있는 백엔드가 0개가 되는 순간이 생김 (완전 다운)
  • 새 프로세스가 1개만(기본 8080) 재기동되어 8081은 영영 안 살아남음

변경 내용

포트(8080, 8081)를 하나씩 순서대로 종료 → 재기동 → 헬스체크 확인하도록 변경 (롤링 재시작). 한쪽이 재시작되는 동안 nginx가 다른 한쪽으로 계속 트래픽을 보내므로 배포 중에도 무중단 유지.

  • 8080: OTel agent 포함 (기존 관측성 유지)
  • 8081: OTel agent 미포함 (메모리 절약, 512MB~1GB 인스턴스에서 발생했던 OOM 이슈 참고)
  • 각 포트 재기동 후 로그의 준비 신호 + /actuator/health UP 여부까지 확인, 실패 시 해당 포트만 실패 처리(다른 포트는 서비스 유지)

테스트

  • 새 인스턴스(t3.small)에서 동일한 2프로세스+nginx 구성으로 실 서비스 검증 완료 (kill -9 강제종료, kill -STOP hang, 전체다운 3가지 실패 케이스 재현 및 확인)
  • CodeDeploy 배포 대상 인스턴스 태그(Name=runnect-prod)를 새 인스턴스로 교체 완료, CodeDeploy 에이전트 설치 및 실행 확인 완료

Summary by CodeRabbit

  • Reliability

    • Deployments now use a rolling process across two application instances, helping maintain service availability during updates.
    • Each instance is verified to start successfully and pass health checks before deployment continues.
    • Failed deployments stop promptly and provide clearer diagnostic logs.
  • Operations

    • Application logs are now separated by instance for easier troubleshooting.
    • Optional OpenTelemetry monitoring support is available during deployment.
    • Nginx is enabled after both instances are confirmed operational.

prod가 nginx+2프로세스(8080/8081) 구조로 전환됨에 따라, 배포 스크립트도
프로세스를 한 번에 재시작하지 않고 포트 하나씩 순서대로 종료→재기동하도록
수정. 재시작 중에도 nginx가 다른 포트로 트래픽을 계속 보내 무중단 배포가
유지된다.
@unam98 unam98 self-assigned this Jul 28, 2026
@coderabbitai

coderabbitai Bot commented Jul 28, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 8dce42a0-1826-4c58-91ee-90a6a79921e7

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • ✅ Review completed - (🔄 Check again to review again)
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/rolling-restart-deploy

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
scripts/deploy.sh (1)

82-88: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Health check substring match can false-positive on nested component status.

grep -q '"status":"UP"' matches anywhere in the response body. With show-details/show-components enabled, Spring Boot Actuator can return an aggregate "status":"DOWN" while individual sub-components still report "status":"UP" (e.g. {"status":"DOWN","details":{"serviceA":{"status":"UP"...), which would still satisfy this grep and be treated as healthy. Whether this app enables detailed health output isn't visible in this diff, but the check should validate the top-level status field specifically rather than any substring match.

♻️ Proposed fix — anchor the match to the top-level field
-  if ! echo "$RESPONSE" | grep -q '"status":"UP"'; then
+  if ! echo "$RESPONSE" | grep -Eq '^\{"status":"UP"'; then

Or, if jq is available in the deployment environment, prefer [ "$(echo "$RESPONSE" | jq -r '.status')" = "UP" ] for a robust parse instead of text matching.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/deploy.sh` around lines 82 - 88, Update the health validation in the
deployment check around the RESPONSE and grep logic to verify the top-level JSON
status field specifically, not any nested component status. Prefer parsing
.status with jq when available; otherwise anchor the text match to the
response’s top-level status while preserving the existing failure return
behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/deploy.sh`:
- Around line 23-30: Update the shutdown flow around OLD_PID in
scripts/deploy.sh to poll until the identified process exits instead of relying
only on a fixed 5-second sleep. After sending SIGTERM with kill -15, wait for
the process to disappear, and add an escalation fallback such as SIGKILL when
the graceful-shutdown timeout is exceeded, before starting the replacement
process.
- Around line 45-51: Reorder the Java command in the deployment startup block so
-Duser.timezone=Asia/Seoul appears before -jar, while keeping $JAR_PATH and the
existing application arguments unchanged.

---

Nitpick comments:
In `@scripts/deploy.sh`:
- Around line 82-88: Update the health validation in the deployment check around
the RESPONSE and grep logic to verify the top-level JSON status field
specifically, not any nested component status. Prefer parsing .status with jq
when available; otherwise anchor the text match to the response’s top-level
status while preserving the existing failure return behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: c19f5068-16f5-413f-a8ee-73869620917c

📥 Commits

Reviewing files that changed from the base of the PR and between ce52d63 and 92e2ce6.

📒 Files selected for processing (1)
  • scripts/deploy.sh

Comment thread scripts/deploy.sh
Comment on lines +23 to +30
OLD_PID=$(pgrep -f "server.port=$PORT" || true)
if [ -n "$OLD_PID" ]; then
echo "> [$PORT] 종료: $OLD_PID"
kill -15 $OLD_PID
sleep 5
else
echo "> [$PORT] 실행 중인 프로세스 없음"
fi

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Fixed 5s sleep after SIGTERM doesn't guarantee the port is free.

After kill -15, the script blindly sleeps 5 seconds with no check that the old process actually exited before starting a new process bound to the same port. If graceful shutdown (e.g., in-flight connection draining) takes longer than 5s, the new java process will fail to bind the port; this is only caught indirectly via the crash-detection loop, causing an avoidable deploy failure for that port.

♻️ Proposed fix — poll for exit with an escalation fallback
   if [ -n "$OLD_PID" ]; then
     echo "> [$PORT] 종료: $OLD_PID"
     kill -15 $OLD_PID
-    sleep 5
+    for i in $(seq 1 10); do
+      kill -0 $OLD_PID 2>/dev/null || break
+      sleep 1
+    done
+    kill -0 $OLD_PID 2>/dev/null && kill -9 $OLD_PID
   else
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
OLD_PID=$(pgrep -f "server.port=$PORT" || true)
if [ -n "$OLD_PID" ]; then
echo "> [$PORT] 종료: $OLD_PID"
kill -15 $OLD_PID
sleep 5
else
echo "> [$PORT] 실행 중인 프로세스 없음"
fi
OLD_PID=$(pgrep -f "server.port=$PORT" || true)
if [ -n "$OLD_PID" ]; then
echo "> [$PORT] 종료: $OLD_PID"
kill -15 $OLD_PID
for i in $(seq 1 10); do
kill -0 $OLD_PID 2>/dev/null || break
sleep 1
done
kill -0 $OLD_PID 2>/dev/null && kill -9 $OLD_PID
else
echo "> [$PORT] 실행 중인 프로세스 없음"
fi
🧰 Tools
🪛 Shellcheck (0.11.0)

[info] 26-26: Double quote to prevent globbing and word splitting.

(SC2086)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/deploy.sh` around lines 23 - 30, Update the shutdown flow around
OLD_PID in scripts/deploy.sh to poll until the identified process exits instead
of relying only on a fixed 5-second sleep. After sending SIGTERM with kill -15,
wait for the process to disappear, and add an escalation fallback such as
SIGKILL when the graceful-shutdown timeout is exceeded, before starting the
replacement process.

Comment thread scripts/deploy.sh
Comment on lines +45 to +51
nohup java $JAVA_AGENT_OPTS -jar \
-Duser.timezone=Asia/Seoul \
$JAR_PATH \
--server.port=$PORT \
>> "$LOG_FILE" 2>&1 &
APP_PID=$!
echo "> [$PORT] 프로세스 시작됨 (PID: $APP_PID)"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== deploy.sh excerpt =="
cat -n scripts/deploy.sh | sed -n '1,130p'

echo
echo "== available java version =="
if command -v java >/dev/null 2>&1; then
  java -version 2>&1 || true
else
  echo "java not installed"
fi

echo
echo "== Java -jar option semantics probe =="
if command -v java >/dev/null 2>&1; then
  mkdir -p /tmp/java-jar-probe
  echo 'public class App { public static void main(String... args) { for(String a: args) System.out.println(a); } }' > /tmp/java-jar-probe/App.java
  javac /tmp/java-jar-probe/App.java
  java -jar -Duser.timezone=Asia/Seoul /tmp/java-jar-probe/App.jar 2>&1 || true
  java -cp /tmp/java-jar-probe -Duser.timezone=Asia/Seoul App 2>&1 || true
  rm -f /tmp/java-jar-probe/App.java /tmp/java-jar-probe/App.class
fi

Repository: Runnect/Runnect-Spring-Boot-Server

Length of output: 1957


🌐 Web query:

Java -jar option JVM options must precede -jar man page

💡 Result:

When launching a Java application using the java command, options intended for the Java Virtual Machine (JVM)—such as -Xmx or -Dproperty=value—must precede the name of the class or the JAR file [1]. The general syntax for the java command is: java [options] classname [args] or java [options] -jar jarfile [args] Everything provided before the class name or JAR file is treated as an option for the JVM, while everything provided after is treated as an argument for the application's main method [1][2]. Regarding specific tools or legacy commands: 1. javaws (Java Web Start): Documentation has explicitly stated that all run-options must precede the JNLP file argument [3]. 2. javadoc: Certain options, such as -locale, must be placed before other options to function correctly [4]. 3. jcmd: Options and arguments for this diagnostic tool have specific syntax requirements, often requiring key or key=value syntax [5]. For most standard Java invocations, the requirement that JVM options precede the application entry point (class or JAR) is a fundamental rule of the Java launcher's command-line parsing [1].

Citations:


🌐 Web query:

Oracle Java man java "If a JAR file is being used, the options must precede the -jar"

💡 Result:

Yes, when using the -jar option with the java command, all command-line options intended for the Java Virtual Machine (JVM)—such as system properties defined with -D—must precede the -jar flag [1][2]. The standard syntax for the java command is [3][4][5]: java [options] -jar jarfile [args...] In this structure: 1. [options]: These are JVM options (e.g., -Dproperty=value, -Xmx, etc.) that must be placed before the -jar argument to be recognized as configuration for the JVM [1][2]. 2. -jar jarfile: This specifies the JAR file to execute. When this flag is used, the JAR file becomes the source of all user classes, and other classpath settings are ignored [3][4][5]. 3. [args...]: Any arguments placed after the JAR file name are treated as application arguments and are passed directly to the main method of the application's entry point class, not to the JVM [3][4][5]. If JVM options (like -D) are placed after the -jar flag, the JVM will typically interpret them as application arguments rather than system settings, which often causes the application to fail to receive the expected configuration [1][2].

Citations:


Critical: Move -Duser.timezone before -jar.

JVM options must precede -jar; with the current order, -jar consumes -Duser.timezone=Asia/Seoul as the jar file name, java fails to launch, and $JAR_PATH becomes an application argument.

Proposed fix
   nohup java $JAVA_AGENT_OPTS -jar \
-    -Duser.timezone=Asia/Seoul \
-    $JAR_PATH \
+    -Duser.timezone=Asia/Seoul \
+    -jar $JAR_PATH \
     --server.port=$PORT \
     >> "$LOG_FILE" 2>&1 &
🧰 Tools
🪛 Shellcheck (0.11.0)

[info] 47-47: Double quote to prevent globbing and word splitting.

(SC2086)


[info] 48-48: Double quote to prevent globbing and word splitting.

(SC2086)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/deploy.sh` around lines 45 - 51, Reorder the Java command in the
deployment startup block so -Duser.timezone=Asia/Seoul appears before -jar,
while keeping $JAR_PATH and the existing application arguments unchanged.

@unam98
unam98 merged commit 8a3216c into main Jul 28, 2026
2 checks passed
@unam98
unam98 deleted the fix/rolling-restart-deploy branch July 28, 2026 16:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants