Hadoop Ecosystem related ports

q29191

　　本文总结了Hadoop生态系统中各个组件使用的端口，包括了HDFS，Map Reduce，HBase，Hive，Spark，WebHCat，Impala，Alluxio，Sqoop等，后续会持续更新。
　　HDFS Ports：

Service

Servers

Default Ports Used

Protocol

Description

Need End User Access?

Configuration Parameters

NameNode WebUI

Master Nodes (NameNode and any back-up NameNodes)

50070

http

Web UI to look at current status of HDFS, explore file system

Yes (Typically admins, Dev/Support teams)

dfs.http.address

50470

https

Secure http service

dfs.https.address

NameNode metadata service

Master Nodes (NameNode and any back-up NameNodes)

8020/9000

IPC

File system metadata operations

Yes (All clients who directly need to interact with the HDFS)

Embedded in URI specified by fs.default.name

DataNode

All Slave Nodes

50075

http

DataNode WebUI to access the status, logs etc.

Yes (Typically admins, Dev/Support teams)

dfs.datanode.http.address

50475

https

Secure http service

dfs.datanode.https.address

50010

Data transfer

dfs.datanode.address

50020

IPC

Metadata operations

No

dfs.datanode.ipc.address

Secondary NameNode

Secondary NameNode and any backup Secondary NameNode

50090

http

Checkpoint for NameNode metadata

No

dfs.secondary.http.address

　　Map Reduce Ports：

Service

Servers

Default Ports Used

Protocol

Description

Need End User Access?

Configuration Parameters

JobTracker  WebUI

Master Nodes (JobTracker Node and any back-up Job­Tracker node )

50030

http

Web UI for JobTracker

Yes

mapred.job.tracker.http.address

JobTracker

Master Nodes (JobTracker Node)

8021

IPC

For job submissions

Yes (All clients who need to submit the MapReduce jobs  including Hive, Hive server, Pig)

Embedded in URI specified by mapred.job.tracker

Task­Tracker Web UI and Shuffle

All Slave Nodes

50060

http

DataNode Web UI to access status, logs, etc.

Yes (Typically admins, Dev/Support teams)

mapred.task.tracker.http.address

History Server WebUI

51111

http

Web UI for Job History

Yes

mapreduce.history.server.http.address

　　HBase Ports：

Service

Servers

Default Ports Used

Protocol

Description

Need End User Access?

Configuration Parameters

HMaster

Master Nodes (HBase Master Node and any back-up HBase Master node)

60000

  

Yes

hbase.master.port

HMaster Info Web UI

Master Nodes (HBase master Node and back up HBase Master node if any)

60010

http

The port for the HBase­Master web UI. Set to -1 if you do not want the info server to run.

Yes

hbase.master.info.port

Region Server

All Slave Nodes

60020

  

Yes (Typically admins, dev/support teams)

hbase.regionserver.port

Region Server

All Slave Nodes

60030

http

Yes (Typically admins, dev/support teams)

hbase.regionserver.info.port

All ZooKeeper Nodes

2888

Port used by ZooKeeper peers to talk to each other.Seehere for more information.

No

hbase.zookeeper.peerport

All ZooKeeper Nodes

3888

Port used by ZooKeeper peers to talk to each other.Seehere for more information.

hbase.zookeeper.leaderport

  

2181

Property from ZooKeeper's config zoo.cfg. The port at which the clients will connect.

hbase.zookeeper.property.clientPort

　　Hive Ports：

Service

Servers

Default Ports Used

Protocol

Description

Need End User Access?

Configuration Parameters

Hive Server2

Hive Server machine (Usually a utility machine)

10000

thrift

Service for programatically (Thrift/JDBC) connecting to Hive

Yes (Clients who need to connect to Hive either programatically or through UI SQL tools that use JDBC)

ENV Variable HIVE_PORT

Hive Metastore

9083

thrift

Yes (Clients that run Hive, Pig and potentially M/R jobs that use HCatalog)

hive.metastore.uris

　　WebHCat Ports：

Service

Servers

Default Ports Used

Protocol

Description

Need End User Access?

WebHCat Server

Any utility machine

50111

http

Web API on top of HCatalog and other Hadoop services

Yes

　　Spark Ports：

Service

Servers

Default Ports Used

Description

Spark GUI

Nodes running spark

7077

Spark web interface for monitoring and troubleshooting

　　Impala Ports：

Service

Servers

Default Ports Used

Description

Impala Daemon

Nodes running impala daemon

21000

Used by transmit commands and receive results by impala-shell

Impala Daemon

Nodes running impala daemon

21050

Used by applications through JDBC

Impala Daemon

Nodes running impala daemon

25000

Impala web interface for monitoring and troubleshooting

Impala StateStore Daemon

Nodes running impala StateStore daemon

25010

StateStore web interface for monitoring and troubleshooting

Impala Catalog Daemon

Nodes running impala catalog daemon

25020

Catalog service web interface for monitoring and troubleshooting

　　Alluxio Ports：

Service

Servers

Default Ports Used

Protocol

Description

Need End User Access?

Alluxio Web GUI

Any utility machine

19999

http

Web GUI to check alluxio status

Yes

Alluxio API

Any utility machine

19998

Tcp

Api to access data on alluxio

No

　　Sqoop Ports：

Service

Servers

Default Ports Used

Description

Sqoop server

Nodes running Sqoop

12000

Used by Sqoop client to access the sqoop server