Ceph 업그레이드 (Jewel -> Luminous)

Ceph Jewel 버전이 설치되어있는 상태에서 최신버전인 octopus까지 올리려 했으나,

Ceph Doc 상에서는 바로 올릴수 없으며, Jewel -> Luminous -> Octopus 순으로 올리라고 되어있습니다.

Doc 상에서 Ceph 안에 데이터가 있을 경우, 망실될 수 있으며, 

데이터 백업 후 진행하는 것을 권장하고 있습니다.

1. OSD 설정 

[root@mgmt ~]# ceph osd set sortbitwise

set sortbitwise

[root@mgmt ~]# ceph osd set noout

set noout

[root@mgmt ~]# ceph -s

    cluster 2a289202-e471-47a5-b672-929818b2bf6b

     health HEALTH_WARN

            noout flag(s) set

     monmap e1: 2 mons at {mgmt=10.5.0.7:6789/0,mon=10.5.0.13:6789/0}

            election epoch 4, quorum 0,1 mgmt,mon

     osdmap e28: 2 osds: 2 up, 2 in

            flags noout,sortbitwise,require_jewel_osds

      pgmap v3143: 192 pgs, 2 pools, 841 MB data, 232 objects

            1893 MB used, 39044 MB / 40937 MB avail

                 192 active+clean

2. 패키지 repo 버전을 변경합니다. (모든 노드에서 변경해야합니다.)

현재 설치되어있는 repo 버전입니다.

[root@mgmt ~]# cat /etc/yum.repos.d/ceph.repo 

[ceph]

name=Ceph packages for $basearch

baseurl=https://download.ceph.com/rpm-jewel/el7/$basearch

enabled=1

gpgcheck=1

priority=1

type=rpm-md

gpgkey=https://download.ceph.com/keys/release.asc

[ceph-noarch]

name=Ceph noarch packages

baseurl=https://download.ceph.com/rpm-jewel/el7/noarch

enabled=1

gpgcheck=1

priority=1

type=rpm-md

gpgkey=https://download.ceph.com/keys/release.asc

[ceph-source]

name=Ceph source packages

baseurl=https://download.ceph.com/rpm-jewel/el7/SRPMS

enabled=0

gpgcheck=1

type=rpm-md

gpgkey=https://download.ceph.com/keys/release.asc

priority=1

Jewel -> Luminous 로 변경

[root@mgmt ~]# cat /etc/yum.repos.d/ceph.repo 

[ceph]

name=Ceph packages for $basearch

baseurl=https://download.ceph.com/rpm-luminous/el7/$basearch

enabled=1

gpgcheck=1

priority=1

type=rpm-md

gpgkey=https://download.ceph.com/keys/release.asc

[ceph-noarch]

name=Ceph noarch packages

baseurl=https://download.ceph.com/rpm-luminous/el7/noarch

enabled=1

gpgcheck=1

priority=1

type=rpm-md

gpgkey=https://download.ceph.com/keys/release.asc

[ceph-source]

name=Ceph source packages

baseurl=https://download.ceph.com/rpm-luminous/el7/SRPMS

enabled=0

gpgcheck=1

type=rpm-md

gpgkey=https://download.ceph.com/keys/release.asc

priority=1

3. 패키지 설치

[root@mgmt ~]# yum install ceph-deploy python-pushy

현재 설치 되어있는 ceph 버전

[root@mgmt ~]# ceph -v

ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)

mgmt 부터 Luminous 버전으로 업그레이드 진행합니다.

[root@mgmt ~]# ceph-deploy install --release luminous mgmt

...

[mgmt][DEBUG ] Complete!

[mgmt][INFO  ] Running command: ceph --version

[mgmt][DEBUG ] ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)

데몬을 재시작해줍니다.

[root@mgmt ~]# systemctl restart ceph-mon.target

[root@mgmt ~]# netstat -nltp

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    

tcp        0      0 10.5.0.7:6800           0.0.0.0:*               LISTEN      16612/ceph-mds      

tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1010/sshd           

tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1059/master         

tcp        0      0 10.5.0.7:6789           0.0.0.0:*               LISTEN      21757/ceph-mon      

tcp6       0      0 :::22                   :::*                    LISTEN      1010/sshd   

[root@mgmt ~]# ceph --version

ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)

[root@mgmt ~]# ceph versions

{

    "mon": {

        "ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 1,

        "unknown": 1

    },

    "mgr": {},

    "osd": {

        "ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)": 2

    },

    "mds": {

        "unknown": 1

    },

    "overall": {

        "ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)": 2,

        "ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 1,

        "unknown": 2

    }

}

나머지 mon서버 하나도 설치 진행해줍니다.

[root@mgmt ~]# ceph-deploy install --release luminous mon

[mon][DEBUG ] Complete!

[mon][INFO  ] Running command: ceph --version

[mon][DEBUG ] ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)

똑같이 데몬을 재시작해줍니다.   

[root@mon ~]# systemctl restart ceph-mon.target

[root@mon ~]# netstat -nltp

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    

tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1030/sshd           

tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1009/master         

tcp        0      0 10.5.0.13:6789          0.0.0.0:*               LISTEN      19064/ceph-mon      

tcp6       0      0 :::22                   :::*                    LISTEN      1030/sshd           

tcp6       0      0 ::1:25                  :::*                    LISTEN      1009/master     

    

[root@mon ~]# ceph -s

  cluster:

    id:     2a289202-e471-47a5-b672-929818b2bf6b

    health: HEALTH_WARN

            noout flag(s) set

 

  services:

    mon: 2 daemons, quorum mgmt,mon

    mgr: no daemons active

    osd: 2 osds: 2 up, 2 in

         flags noout

 

  data:

    pools:   2 pools, 192 pgs

    objects: 232 objects, 842MiB

    usage:   1.85GiB used, 38.1GiB / 40.0GiB avail

    pgs:     192 active+clean

 

여기까지 mon 데몬들은 다 업그레이드가 완료되었으며,

Jewel 버전에서 없던 MGR 데몬을 새로 설치해줍니다. 

[root@mgmt ~]# ceph-deploy mgr create mgmt mon

[root@mgmt ~]# systemctl restart ceph-mgr.target

mds도 데몬 재시작 해줍니다.

[root@mgmt ~]# systemctl restart ceph-mds.target

[root@mgmt ~]# netstat -nltp

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    

tcp        0      0 10.5.0.7:6800           0.0.0.0:*               LISTEN      22295/ceph-mds      

tcp        0      0 10.5.0.7:6801           0.0.0.0:*               LISTEN      22120/ceph-mgr      

tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1010/sshd           

tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1059/master         

tcp        0      0 10.5.0.7:6789           0.0.0.0:*               LISTEN      21757/ceph-mon      

tcp6       0      0 :::22                   :::*                    LISTEN      1010/sshd           

[root@mgmt ~]# ceph versions

{

    "mon": {

        "ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 2

    },

    "mgr": {

        "ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 2

    },

    "osd": {

        "ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)": 2

    },

    "mds": {

        "ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 1

    },

    "overall": {

        "ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)": 2,

        "ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 5

    }

}

osd도 동일하게 버전 업그레이드 진행해주며, 데몬 재시작 해줍니다.

[root@mgmt ~]# ceph-deploy install --release luminous osd0 osd1

[root@osd0 ~]# systemctl restart ceph-osd.target

[root@osd1 ~]# systemctl restart ceph-osd.target

[root@mgmt ~]# ceph versions

{

    "mon": {

        "ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 2

    },

    "mgr": {

        "ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 2

    },

    "osd": {

        "ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 2

    },

    "mds": {

        "ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 1

    },

    "overall": {

        "ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 7

    }

}

[root@mgmt ~]# ceph osd require-osd-release luminous

[root@mgmt ~]# ceph osd unset sortbitwise

sortbitwise is unset

[root@mgmt ~]# ceph osd unset noout

noout is unset

 

참고글)

https://www.virtualtothecore.com/upgrade-ceph-cluster-luminous/

https://www.programmersought.com/article/7678699958/