https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LUDOC-311 Lustre Documentation <p>现在lustre文件系统上构建大数据spark集群（docker集群），遇到spark集群无法访问lustre文件系统的情况，恳请帮助解决。<br/> 当前环境：<br/> 内核 3.10.0-229.7.2.el7.x86_64<br/> docker镜像用centos制作，镜像安装了lustre文件系统的客户端<br/> lustre-client-modules-2.5.37.7-3.10.0_229.7.2.el7.x86_64.x86_64.rpm<br/> lustre-client-2.5.37.7-3.10.0_229.7.2.el7.x86_64.x86_64.rpm<br/> 镜像中封装了hadoop2.5.2和spark1.4.1<br/> docker 集群启动 通过-v /mnt:/mnt 挂载lustre文件系统<br/> 在master容器中成功启动hadoop集群和spark集群，<br/> 通过jps在master容器中看见NodeManager、Master、ResourceManager、Worke<br/> slave容器中看见Worker和NodeManager<br/> 运行情况：<br/> hadoop集群运行pi正常，运行wordcount也正常<br/> 容器中运行./spark-shell可以正常访问lustre文件系统</p> <p> 问题：<br/> master容器运行 ./spark-shell --master spark://master:7077可以正常启动<br/> 但是访问luster文件系统报错，错误见附件</p> 当前环境：<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;内核 3.10.0-229.7.2.el7.x86_64<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;docker镜像用centos制作，镜像安装了lustre文件系统的客户端<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;lustre-client-modules-2.5.37.7-3.10.0_229.7.2.el7.x86_64.x86_64.rpm<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;lustre-client-2.5.37.7-3.10.0_229.7.2.el7.x86_64.x86_64.rpm<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;镜像中封装了hadoop2.5.2和spark1.4.1<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;docker 集群启动 通过-v /mnt:/mnt 挂载lustre文件系统<br/> LUDOC-311

Spark is not able to access Lustre within Docker

Question/Request Major Closed Incomplete Jodi Levi yale wang build Wed, 28 Oct 2015 00:56:34 +0000 Thu, 29 Oct 2015 00:55:51 +0000 Thu, 29 Oct 2015 00:55:51 +0000 0 2 <p>Yale re-filed this issue to LU-7348, so let me close this one.</p> Development Epic server Epic/Theme test Project Rank 1|hzxrmv: Rank (Obsolete) 9223372036854775807