How To Change A Failed Disk For Hdfs

Failed Disk Replacement with Navigator Encrypt

Hardware fails.  Especially hard disks.  Your Hadoop cluster volition be operating with less capacity until that failed disk is replaced.  Using full disk encryption adds to the replacement trouble.  Hither is how to do it without bringing down the entire machine (assuming of course that your disk is hot swappable).

Assumptions:

• Cloudera Hadoop and/or Cloudera Kafka environment.
• Cloudera Manager is in apply.
• Cloudera Navigator Encrypt is in apply.
• Physical hardware that will allow for a data disk to exist hot swapped without powering down the entire auto. Otherwise y'all can pretty much skip steps 2 and 4.
• We are replacing a data deejay and non an Os disk.

Steps:

The following are steps to replace a failed disk that is encrypted past Cloudera Navigator Encrypt.   If any of the settings are missing in your Cloudera Director (CM), you might consider upgrading CM to a newer version.

1. Determine the failed deejay.  The case used here is a deejay that is mounted at /information/0.
2. Configure data directories to remove the disk you lot are swapping out:
   1. HDFS
      1. Go to the HDFS service.
      2. Click the Instances  tab.
      3. Click the affected DataNode.
      4. Click the Configuration  tab.
      5. Select .

      6. Modify the value of the DataNode Information Directory  belongings to remove the directories that are mount points for the deejay you are removing.

         Warning: Change the value of this property only for the specific DataNode example where yous are planning to hot bandy the deejay.Exercise not edit the role grouping value for this property. Doing so volition crusade data loss.

      7. Click Salvage Changes  to commit the changes.
      8. Refresh the affected DataNode. Select .
   2. YARN
      1. Go to the YARN service.
      2. Click the Instances  tab.
      3. Click the affected NodeManager.
      4. Click the Configuration  tab.
      5. Select .

      6. Alter the value of the NodeManager Local Directories  property to remove the directories that are mount points for the disk yous are removing.

         Warning: Change the value of this holding only for the specific NodeManager instance where you are planning to hot swap the disk.Practice not edit the function group value for this property. Doing and then volition cause information loss.

      7. Change the value of the NodeManager Container Log Directories  holding to remove the directories that are mountain points for the deejay you are removing.

         Alarm: Change the value of this property but for the specific NodeManager instance where you are planning to hot swap the deejay.Do not edit the role group value for this property. Doing so will cause data loss.

      8. Click Save Changes  to commit the changes.
      9. Refresh the affected NodeManager. Select .
   3. Impala
      1. Go to the Impala service.
      2. Click the Instances  tab.
      3. Click the affected Impala Daemon.
      4. Click the Configuration  tab.
      5. Select .

      6. Change the value of the Impala Daemon Scratch Directories  holding to remove the directories that are mountain points for the disk you are removing.

         Alert: Modify the value of this property merely for the specific Impala Daemon instance where you are planning to hot swap the deejay.Practise non edit the role grouping value for this belongings. Doing so volition cause data loss.

      7. Click Salvage Changes  to commit the changes.
      8. Refresh the affected Impala Daemon. Select .
   4. Kafka
      1. Go to the Kafka service.
      2. Click the Instances  tab.
      3. Click the affected Kafka Banker.
      4. Click the Configuration  tab.
      5. Select .

      6. Change the value of the  Log Directories  property to remove the directories that are mountain points for the disk you are removing.

         Alarm: Alter the value of this property just for the specific Kafka Banker instance where you are planning to hot swap the disk.Practice not edit the role group value for this property. Doing then volition cause data loss.

      7. Click Save Changes  to commit the changes.
      8. Refresh the affected Kafka Broker. Select .
3. Remove the quondam deejay and add the replacement disk.
   1. List out the disks in the system, taking note of the name of the failed disk. (lsblk; lsscsi)
   2. Determine the failed disk.  Instance used here is /information/0 which is mounted at /navencrypt/0.  (readlink -f /information/0)
   3. Make up one's mind the Navigator Encrypt DISKID of the failed source device. (grep /navencrypt/0 /etc/navencrypt/ztab)
   4. Make clean up Navigator Encrypt entries. (navencrypt-fix --undo ${DISKID} || navencrypt-prepare --undo-force ${DISKID})
      1. Also possibly demand to use: (cryptsetup luksClose /dev/mapper/0; dd if=/dev/zero of=${DISK}one ibs=1M count=1)
   5. Remove failed disk.
   6. Add replacement disk.
   7. Perform whatever HBA configuration (i.east. Dell PERC/HP SmartArray RAID0 machinations).
   8. Determine the name of the new disk.  Example used here is /dev/sdo. (lsblk; lsscsi)
   9. Partition the replacement disk. (parted -southward ${DISK} mklabel gpt mkpart primary xfs ane 100%)
   10. Accept Navigator Encrypt configure the disk for encryption and write out a new filesystem. (navencrypt-gear up -t xfs -o noatime --use-uuid ${DISK}i /navencrypt/0)
   11. Fix the symlink target directory installed by navencrypt-motility. (mkdir -p $(readlink -f /data/0))
4. Configure data directories to restore the disk you have swapped in:
   1. HDFS
      1. Change the value of the DataNode Data Directory  property to add dorsum the directory that is the mount signal for the disk you added.
      2. Click Relieve Changes  to commit the changes.
      3. Refresh the affected DataNode. Select .
      4. Run the HDFS fsck utility to validate the health of HDFS.
   2. YARN
      1. Change the value of the NodeManager Local Directories  and NodeManager Container Log Directories properties to add back the directory that is the mount point for the disk you added.
      2. Click Relieve Changes  to commit the changes.
      3. Refresh the affected DataNode. Select .
   3. Impala
      1. Change the value of the Impala Daemon Scratch Directories  property to add together dorsum the directory that is the mount point for the disk you added.
      2. Click Salve Changes  to commit the changes.
      3. Refresh the affected DataNode. Select .
   4. Kafka
      1. Change the value of the Log Directories  holding to add together back the directory that is the mount bespeak for the disk yous added.
      2. Click Save Changes  to commit the changes.
      3. Refresh the affected Kafka Broker. Select .

Reference Links:

https://www.cloudera.com/documentation/enterprise/latest/topics/admin_dn_swap.html

https://world wide web.cloudera.com/documentation/enterprise/latest/topics/navigator_encrypt_prepare.html#concept_device_uuids

Source: https://intelligentsysadmin.wordpress.com/2018/08/24/failed-disk-replacement-with-navigator-encrypt/

Posted by: mathewssuraing.blogspot.com

Share this post