EntryDescriptionNo95 < Main

Tags: , view all tags
---+ Instalação do dcache nos Thors SUN X4540

---++Description
Acrescentando storage para o Dcache utilizando o Thor X4540 da Sun
 
---++Cabeamento dos Thors

|                             |THOR1|THOR2|THOR3|
|||||
|thor port Net0 Eth1         |cx7.49|cx7.50|cx7.51|
|porta Switch Sprace  |25       |26     |27     |
|                             |cx6.41|cx6.42|cx6.43|
|||||
|thor port NetMng               |cx7.52|cx7.53|cx7.54|
|porta Switch Mng GridUnesp|15      |16      |17     |
|                                      |cx1.73|cx1.74|cx1.75|
|||||
|thor port Net1 Eth2          |cx1.05|cx1.06|cx1.07|
|porta Switch Sprace  |7       |8     |9     |

---++Endereços IP e hostnames

Qual o endereço inicial do Thor ??? ----> Acesse o console serial para configurar o IP

login e senha (dãããã)

<pre>cd /SP/network
ls
set pendingipaddress=192.168.2.161
set pendinggateway=192.168.2.150
set commit pending=true
</pre>
 
Fisicamente os Thor estão ordenados de baixo para cima, logo, o 1o thor de baixo é o No 1.

|THOR|hostname|End. IP Net0|End. IP ILOM| End. IP Hexad|End. Externo|
|1|spstrg01|192.168.1.161|192.168.2.161|C0.A8.01.A1|200.136.80.11|
|2|spstrg02|192.168.1.162|192.168.2.162|C0.A8.01.A2|200.136.80.12|
|3|spstrg03|192.168.1.163|192.168.2.163|C0.A8.01.A3|200.136.80.13|

---++Kickstart 

/tftpboot/pxelinux.cfg<br>

 thor-install

[allan@spserv01 pxelinux.cfg]$ cat thor-install<br>
default CentOS-5.4-install/vmlinuz<br>
append initrd=CentOS-5.4-install/initrd.img method=http://200.145.46.3/CentOS ks=http://200.145.46.3/thor_sprace.ks ksdevice=eth1

IP decimal  --->   IP Hexedecimal<br>
192.168.1.161 = C0A801A1 <br>
192.168.1.162 = C0A801A2<br>
192.168.1.163 = C0A801A3<br>

CentOS-5.4-install <br>

o ks utilizado para os thor está em http://200.145.46.3/thor_sprace.ks 

 ls -l<br>
lrwxrwxrwx 1 root root  12 Mar 10 16:53 C0A801A1 -> thor-install<br>
lrwxrwxrwx 1 root root  12 Mar 10 16:54 C0A801A2 -> thor-install<br>
lrwxrwxrwx 1 root root  12 Mar 10 16:54 C0A801A3 -> thor-install<br>
-rw-r--r-- 1 root root 525 Jan  7 09:36 default<br>
-rw-r--r-- 1 root root 166 Mar 11 14:46 thor-install


---++Criação dos RAIDs de dados

   * Criar o layout básico dos discos

<pre>
echo "
n
p
2


t
2
fd
w
" > fdisk.expect
</pre>

   * Executar o fdisk:

<pre>
for i in a b c d e f g h i j k l m n o p q r s t u v w x y z;do fdisk \
/dev/sd$i < fdisk.expect; done

for i in a b c d e f g h i j k l m n o p q r s t u v w x y z;do fdisk \
/dev/sda$i < fdisk.expect; done
</pre>

   * Depois de criar as partições devemos reiniciar a máquina

<pre>reboot
</pre>

   * Criar os RAIDs

<pre>
mdadm --create /dev/md10 --force --chunk=128 --level=5 --raid-devices=6 --spare-devices=0 /dev/sda2 /dev/sdi2 /dev/sdq2 /dev/sdy2 /dev/sdag2 /dev/sdao2

mdadm --create /dev/md11 --force --chunk=128 --level=5 --raid-devices=6 --spare-devices=0 /dev/sdb2 /dev/sdj2 /dev/sdr2 /dev/sdz2 /dev/sdah2 /dev/sdap2

mdadm --create /dev/md12 --force --chunk=128 --level=5 --raid-devices=6 --spare-devices=0 /dev/sdc2 /dev/sdk2 /dev/sds2 /dev/sdaa2 /dev/sdai2 /dev/sdaq2

mdadm --create /dev/md13 --force --chunk=128 --level=5 --raid-devices=6 --spare-devices=0 /dev/sdd2 /dev/sdl2 /dev/sdt2 /dev/sdab2 /dev/sdaj2 /dev/sdar2

mdadm --create /dev/md14 --force --chunk=128 --level=5 --raid-devices=6 --spare-devices=0 /dev/sde2 /dev/sdm2 /dev/sdu2 /dev/sdac2 /dev/sdak2 /dev/sdas2

mdadm --create /dev/md15 --force --chunk=128 --level=5 --raid-devices=6 --spare-devices=0 /dev/sdf2 /dev/sdn2 /dev/sdv2 /dev/sdad2 /dev/sdal2 /dev/sdat2

mdadm --create /dev/md16 --force --chunk=128 --level=5 --raid-devices=6 --spare-devices=0 /dev/sdg2 /dev/sdo2 /dev/sdw2 /dev/sdae2 /dev/sdam2 /dev/sdau2

mdadm --create /dev/md17 --force --chunk=128 --level=5 --raid-devices=6 --spare-devices=0 /dev/sdh2 /dev/sdp2 /dev/sdx2 /dev/sdaf2 /dev/sdan2 /dev/sdav2
</pre>

---++Benchmarking
<pre>
rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm

yum install  xfsprogs bonnie++ 

mkfs.xfs /dev/md10 

mount /dev/md10 /mnt 
cd /mnt
mkdir teste
chown allan.allan teste
cd teste/
bonnie++ -f -n 0 -u 7833 -s 131072:131072
</pre>

O 7833 é o gid para allan

Alguns comandos mdam:

mdadm --stop /dev/md10 para desmontar o raid ( antes, desmontar o ponto de montagem)

* Test A: Single disk
      * no RAID
<pre>rmount /dev/sdaa2 /mnt
</pre>


   * Test B: RAID5 (6 disks)
      * One disk from each controller
<pre>mdadm --create /dev/md10 --force --chunk=128 --level=0 --raid-devices=6 --spare-devices=0 /dev/sdc2 /dev/sdk2 /dev/sds2 /dev/sdaa2 /dev/sdai2 /dev/sdaq2
</pre>
   * Test C: RAID50 (48 disks)
      * A single RAID 0 with 8 RAID5, each one with 6 disks

<pre>mdadm --create /dev/md20 --force --chunk=512 --level=0 --raid-devices=8 --spare-devices=0 /dev/dm10 /dev/md11 /dev/md12 /dev/md13 /dev/dm14 /dev/md15 /dev/dm16 /dev/md17
</pre>

   * Test D: RAID0 (6 disks)
      * One disk from each controller

   * Test E: RAID0 (24 disks)
      * 4 disks from each controller

   * Test F: RAID0 (48 disks)
      * all disks 
<pre>mdadm --create /dev/md10 --force --chunk=128 -e 1 --level=0 --raid-devices=48 /dev/sda2 /dev/sdaa2 /dev/sdab2 /dev/sdac2 /dev/sdad2 /dev/sdae2 /dev/sdaf2 /dev/sdag2 /dev/sdah2 /dev/sdai2 /dev/sdaj2 /dev/sdak2 /dev/sdal2 /dev/sdam2 /dev/sdan2 /dev/sdao2 /dev/sdap2 /dev/sdaq2 /dev/sdar2 /dev/sdas2 /dev/sdat2 /dev/sdau2 /dev/sdav2 /dev/sdb2 /dev/sdc2 /dev/sdd2 /dev/sde2 /dev/sdf2 /dev/sdg2 /dev/sdh2 /dev/sdi2 /dev/sdj2 /dev/sdk2 /dev/sdl2 /dev/sdm2 /dev/sdn2 /dev/sdo2 /dev/sdp2 /dev/sdq2 /dev/sdr2 /dev/sds2 /dev/sdt2 /dev/sdu2 /dev/sdv2 /dev/sdw2 /dev/sdx2 /dev/sdy2 /dev/sdz2
</pre>

---++ Resultados

|* test* | *sequential write (CPU)* | *sequential read (CPU)* | *random seek (CPU)* |
| A |     85MB/s (18%) |     87MB/s (11%) |  195.3MB/s (3%) |
| B |   82MB/s (25%) |    260MB/s (49%) |  318.6MB/s (9%) |
| C |    247MB/s (48%) |    130MB/s (45%) |    354.6MB/s (11%) |
| D |    292MB/s (29%) |    325MB/s ( 40%) |  323.7MB/s (6%) |
| E |    963MB/s (85%) |    889MB/s (42%) |   377.7MB/s (7%) |
| F |    733MB/s (65%) |   1153MB/s (88%) |   391.5MB/s (7%) |


---++ Instalação do software para os pools:

O primeiro passo é instalar  a versão do Java *JDK*, do site de SUN. 
Crie os diretórios:
<pre>
mkdir /etc/grid-security
mkdir /etc/grid-security/certificates
</pre>
Adicionar a seguinte linha ao =/etc/fstab=
<pre>
osgce:/opt/osg-1.2.4/globus/TRUSTED_CA  /etc/grid-security/certificates           nfs     rw,auto,hard,bg,rsize=32768,wsize=32768,udp,nfsvers=3
</pre>

O certificado para esta máquina foi solicitado à partir da osg-ce:
<pre>
. /OSG/setup.sh
mkdir spstrg02
cd  spstrg02
cert-gridadmin -host spstrg02.sprace.org.br -prefix spstrg02 ca doegrids -affiliation osg -vo dosar -show -email mdias@ift.unesp.br
scp spstrg02* spstrg02.sprace.org.br:/tmp/.
</pre>
Retorne à =spstrg02= e instale os certificados de máquina:
<pre>
mv /tmp/spstrg02cert.pem /etc/grid-security/hostcert.pem
mv /tmp/spstrg02key.pem /etc/grid-security/hostkey.pem
chown root: /etc/grid-security/hostcert.pem /etc/grid-security/hostkey.pem
chmod 400 /etc/grid-security/hostkey.pem
chmod 444 /etc/grid-security/hostcert.pem
openssl x509 -text -noout -in /etc/grid-security/hostcert.pem
</pre>

Faça o download do pacote dCache-server
<pre>
cd /tmp
wget http://www.dcache.org/downloads/1.9/dcache-server-1.9.5-9.noarch.rpm
</pre>
Configuração:
<pre>
cp /opt/d-cache/etc/dCacheSetup.template /opt/d-cache/config/dCacheSetup
</pre>
As modificações feitas no arquivo acima foram:
<pre>
serviceLocatorHost=osg-se.sprace.org.br
java="/usr/bin/java"
useGPlazmaAuthorizationModule=true
useGPlazmaAuthorizationCell=false
performanceMarkerPeriod=10
</pre>
Outro arquivo de configuração:
<pre>
cp /opt/d-cache/etc/node_config.template /opt/d-cache/etc/node_config
</pre>
alterando:
<pre>
vim /opt/d-cache/etc/node_config
SERVER_ID=sprace.org.br
NAMESPACE_NODE=osg-se.sprace.org.br
NODE_TYPE=pool
SERVICES=gridftp dcap gsidcap
</pre>
Edite também:
<pre>
vim /opt/d-cache/etc/dcachesrm-gplazma.policy
saml-vo-mapping="ON"
kpwd="ON"
saml-vo-mapping-priority="1"
kpwd-priority="2"
mappingServiceUrl="https://spserv01.sprace.org.br:8443/gums/services/GUMSAuthorizationServicePort"
</pre>
Comente a  linha iniciada com  =XACMLmappingServiceUrl=. De um pool já em operação, copie os seguintes arquivos:
<pre>
scp /etc/grid-security/storage-authzdb spstrg02:/tmp
scp /opt/d-cache/etc/dcache.kpwd spstrg02:/tmp
</pre>
e mova na spstrg02 para os lugares adequados
<pre>
mv /tmp/storage-authzdb /etc/grid-security/storage-authzdb
mv /tmp/dcache.kpw /opt/d-cache/etc/dcache.kpwd
</pre>
Adicione a seguinte linha (se ela não estiver lá) ao começo do último arquivo:
<pre>
version 2.1
</pre>
Rode o script para instalação
<pre>
/opt/d-cache/install/install.sh
</pre>


A preparação dos pools é feita da seguinte forma, formatando os pools e preparando os sistemas de arquivos (verifique se o pacote = xfsprogs= está instalado):
<pre>
/sbin/mkfs.xfs /dev/md1X
mkdir /raid{0,1,2,3,4,5,6,7}
for i in `seq 0 7`;  do mount /dev/md1$i /raid$i;done
/opt/d-cache/bin/dcache pool create 4654G /raid1/pool1
</pre>
e assim vão sendo criados os pools =/raidX/poo1= . Adicione os pools ao dCache:
<pre>
 /opt/d-cache/bin/dcache pool add spstrg02_1 /raid0/pool1/
/opt/d-cache/bin/dcache pool ls
</pre>
e inicie:
<pre>
/opt/d-cache/bin/dcache start
</pre>
-- Main.AllanSzu - 12 Mar 2010
Topic revision: r11 - 2010-03-19 - MarcoAndreFerreiraDias
antalya escort bursa escort eskisehir escort istanbul escort izmir escort