EntryDescriptionNo10 < Main

Tags: , view all tags
---+ adm.cluster e storage.grid down
---++Description

07h25  e o Ganglia acusa a adm.cluster e a storage.grid down 05h15 min atras. O load da spg00 bateu em 318! O Nara acusa um pico de tr'afego das 05h ate as 06 de 30Mb/s. GFTP spraid,sRM-spraid e spraid_# offline para  dcache.
<pre>
[root@sprace:root]#ping spraid
PING spraid.if.usp.br (200.136.80.5) from 200.136.80.3 : 56(84) bytes of data.
64 bytes from spraid.if.usp.br (200.136.80.5): icmp_seq=1 ttl=64 time=1.09 ms
64 bytes from spraid.if.usp.br (200.136.80.5): icmp_seq=2 ttl=64 time=0.173 ms
--- spraid.if.usp.br ping statistics ---
2 packets transmitted, 2 received, 0% loss, time 1004ms
rtt min/avg/max/mdev = 0.173/0.634/1.095/0.461 ms
 </pre> 
Mas nao responde ssh. Estamos UP para o resto.  <pre> condor_status</pre> na spgrid acusa todas as maquinas ok. A SPRace esta ok (estou digitando nela).
---++Updates

07h53 . A SPRaid voltou ao ar. Infelizmente me loguei no console mas foi inutil: a quantidade de logs <pre> Out of memory: process java ...</pre> impossibilitaram qualquer operacao em qualquer terminal, entao rebootei na unha e o <pre>fsck</pre> deu conta do recado.
O <pre>ntpd</pre> nao subiu OK
Agora, no sprace
<pre>
[root@spraid root]# /opt/d-cache/bin/dcache-core start
/pnfs/if.usp.br/ not mounted - going to mount it now ...
Starting dcache services:
Starting gridftp-spraidDomain  6 5 4 3 2 1 0 Done (pid=1133)
Starting srm-spraidDomain  6 5 4 3 2 1 0 Done (pid=1223)

[root@spraid root]# /opt/d-cache/bin/dcache-pool start

Starting dcache pool: Starting spraidDomain  6 5 4 3 2 1 0 Done (pid=1400)

[root@spraid root]# /etc/init.d/ntpd restart
Shutting down ntpd:                                        [  OK  ]
ntpd: Synchronizing with time server:                      [  OK  ]
Starting ntpd:                                             [  OK  ]
</pre>
Demorou mas o  monitoramento  em 
http://spdc00.if.usp.br:2288/cellInfo
acusou ok para os spraid_1, spraid_2, etc

---+++Fulano em dd/mm/aaaa
Topic revision: r1 - 2006-10-06 - MarcoAndreFerreiraDias
antalya escort bursa escort eskisehir escort istanbul escort izmir escort