Apache Spark源码走读之19

hc6538

　　欢迎转载，转载请注明出处，徽沪一郎。

概要
　　本文主要讲述在standalone cluster部署模式下，Spark Application在整个运行期间，资源(主要是cpu core和内存)的申请与释放。
　　构成Standalone cluster部署模式的四大组成部件如下图所示，分别为Master, worker, executor和driver，它们各自运行于独立的JVM进程。
http://www.iyunv.com/hseagle/p/data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAbcAAADcCAIAAAB8sXiRAAAAA3NCSVQICAjb4U/gAAAACXBIWXMAAA7EAAAOxAGVKw4bAAAgAElEQVR4nO3dd1wT9/8H8Hcg7L0jq6CoOEAFV0HrCi5w1lIBrdZViwKO9qdV6/yqVWtbR617ExStFhXQAi7AVVFZIgISdiBsEiDkkvz+OBtThAgRSAjv58M/yCd3ySfj8/J9d5/cUUQiEXRhcXFx69evj4mJoVKpH1xYIBAsXbp0165dxsbGHdA3hD5ebGwsl8udOHFio/bz588PGDDAycmpybVSUlKuXbs2Z84cW1vb9u+joqN08ZRECCHpVOTdAYQQUmiYkgghJM2Hd8aJUSiU9uuHfOFuh/amxF8eSfhFUkqtSEkAIF5eaKd+yBG172x5d6FL4F3PlHcX2pfGFAd5dwG1C9ziRgghaTAlEUJIGkxJhBCSBlMSIYSkUayUrOHW5RSw5d0LhBB6p3XHuJt0ITw+M5clEAjfPaiqqraWOs3UsE8Pa6detqqqLc1i+vxtL14xk6//3MvO8uM7hhTf2eg/c0oKAEQA8PmIyX1teza35MO0hOjncQCgoqLqbO84ZRi943qJurY2SMnZnu5v8oqdp35Xz+N/Pn6Yh7uzoZ4Ou6I6LObp/B8OmRrprV8681uf8S2ZMdfXwbqimmuop/PxvUKdwlf0z+NTn45dOxsAkrNfha7/o7klVxzZ+iIr1VBHP+FguLVpt498XpFI1EWmcKKP1zZb3N1tLBxsaQCw2Ju+6ItxsyYO/9Zn/IVfV7z4a7ehnk7g/05N998jFAo/+Dindvq/vrXP3MSgTXqFOgX3foNtzSwpFMq1x9Hp+W+aXOZO4sPENy8BwMne8eMjEgA2nt1b38D7+MdBXUGb7ZdUV2uiLO3Twzrm9EZzE4Pwe89+3HexrZ6rwxAEIe8udCYyv12a6hpTh9FFItEvV441ucCvV49NG+4BAJpqGrL3718FZazDEec//nFQF9HuR29oZoZbA70BYO+pG/msMnE7n0/w+QQANDQQkr/rqq3jyfAzL4IQVFRx3l+xuWf5IC6Xu3Xr1sePH7e2J11ZWlqar68vk8mUYd2g6QtVVVQZd8KKyksa3ZWa8zqjgDndbUJz6zbwG5orDAUCQR2vHgB4/LcL1NbXfbH92+paTpPLV9fWNPlQ3Ppa8g+sQLugjjjG7es1gkpVJQjB5VuPACA7v2TdLyG2Y/xvxSfuPxthOGR+38mruLX1D1+8nrf2d7NPF1XV1ALA0YvRbrM3DJ31w+i5m8+H3Scf6kUac8G6P9xnb2BcjyNbsvNL/L7bP2P5nuFfrrf6bOnvwTfF7U0+ywd7S+ajrq4uALi7u7fHG6KsnJyc3Nzc7O3tZcjKTyysPh8xqYFoOBB2qtFdv1w5Fjjta1UV1ffXCrkbNup774A/No1Z82W/JeMux4aL76qurVm8b43/7xsC/tg46ntv1+WehIDILSmcsGHu86xUAJj84zz6D77xqU/J5Y9EBE/+cd7sncstfFymbFqQU1IAAISAuBJ/c+KGuR7r/ArKWMNWTDOc1f9YZEirXhrq7DoiJbW1NPo5WANAQuobPp+48yjlVtwLdnn1jTvP2BU1rv26F7LL07MLUzPy/op+wmvgk2st+ZK++Itxz15m62hpzJn2Gdk4sI/d8AE9x7k5+U4ZAQB5RaWeS3b+6P/59cNrX4b/OtLVMWj76T9C/m7uWbh10goBcT5u2rSJTqevW7eund8YJbR8+XI6nR4SEiJDVq6euRgAjkaGVHFrxI2FZcXRz+Pm0We9v3zk07vz965eN3vZkcCd8XuvaGlozf/lu7LqCvLeH07t6mfb60jgzuMrdp9YubugrBgAbM0tY3++PMppGABEbDsTvZPh3m8wAPx45ucUZvqNLacitp25/dOFO0kPx6/z49Rxn6QnpuVl3kl8yCeIrcH7PIeM0VBTz2blyv4GoU6og+ZLmhnpA0B5JUdNjbpg1tiZHsMAgEKBbUFfxjK2suKOufTrvuiLcYP62Euu9dX0UTbdTB4lZtTVN4gbr/z9OGDOJPLvtXsZn08Y7tjdCgBUVVUC5k4CgJ1Hrjb3LM0dF5LMR7IlOjpaTU2NglovOjqafA9bm5UDe/QbO8Ctpo5zNJIhbjx47fTCCV9qa2q9v/z95EcAYGNmCQAqKiru/QbzCX5WUQ55b8zz+ApOFfm3g6XdnLEzmnve9Pw3+8JO7lrwg4qKCgC49nQa0W8wszj/4v3rbn1d189erqqi+rrgzVpv/41+K1iMhB1fr2nJy0FKow1mArVEfQMfAPR03n7XNdSpAOA52oW8qaWp/vYPDXXJtVRVVb750mPDbxcYN+IWzhoLAC8z881M9M2M9QGAzyfCYv4Z5txz8Y9HyOW5tfWW5kYAUFHFMTLQbe5Z3rd48eKQENyMahchISGpqaktXHj150tuJz44EHYqcNp8DTWNmlrO+dtXEw6GN7nwiumLhvYeRE6xzC0pzCnOB4n9hib6Rrsu/SEUCr//4ht9bb31PgFNbrMDwJ9xkepU9ZVHt4pbqrg1lsYWeexC8qa6mpqjtYM9zQYAtDQ0W/hakNLooJRkFrABoKfd2zkclGamqr3fvOiLsVt/v3yIcYtMySMXo5bOHk/elVNYWs/jL/h8jN/Ukc08WksnxDEYjB07dqxbt06cleHh4ZMnT27h6khScnKys7Mz+bezs/OhQ4fc3d1b+FnQB41wtu+TlJ12/vbVhRNmH791YeqnHmYGJk0ubGFkOsNtQmzKk6ORDAtDU30dPQAQwdtjdDu+/r+Z277Zffnw0UhG4PQFq2Ysaq4P6flZZgbGRwJ3NtcrCuDMyi6tI7a4X70pII9uTxw5sLXrmhrpfznZLfFVTvyz9No6XnJ6rrtLb/KuBj4BAOnMwjbppJ2dHYPByM7O9vHxAQBPT082G38r2WpcLnf8+PEA4OzsHBcXl5iY2NojYOTeyV+vHOfxeYeunw2avrDZ56qvnbM7aM/lw78s+fHnxRvsLWwk7x3lNPz5wcjZo6ZW1dZsDf7t05XT2VVlTT4On+DnsgvJQ+EIva8jUnLXsTAAGOHqKA64VlnmNwEAfg++eSE83sfr3ZAjN64vRjwg5/qI3bib0Kil5SSzkk6n43zJ1po+fbq5ubls+UiaNXKyrZllRiFz3s+rXBz697S0a27JwD82RT69G7L2YJPFZlZRjq255Znvfnmy79rAHv3S8jJ/unioycexNLEgBETo/RuSjQKB4NqjKBn6j5RPm6WksJnZiKev3D0Xdt/IQOfkjm9le+TB/XsMc3a4EvXk+OXbvl4jxO2G+jpDnR2ycosDt58S/7CHXV59MeKBWlNT3FuOzMqwsDCcL9kqycnJmzdvliEfhUKhUPj2+0NVpQZOXwAAVx/cWjVzseQyACAUvfsF143HMQbaejqa2uRNcj6seFbsT6FvM9HZvk/ktjPaGlrM4nyyRYWiAgCE4O1/gR4uIwFg7amfUpjp4gffF3ZSW6OJQ0aoC2qz/ZLFpVUgsVcIAIrYFbuOhR08f9Opl23obyu721iI76qt54lXkVTDrQcAbh3PUP8/P+Ve5jfxqzUHhzj10NH+z77zbUFfTly041hozOPEjGnjhnDreKGRDxl7A6U/SwvZ2dnZ2dnJtm7X1NzFnaUTiUSl1RWl1eW25m9PcbJgvPf2kAN9bB2GOQ4SL1ZSVQYArIp3u0GsTGmpOa8PXjsza+TkS7Hh1x5HA8DDtGfPs1JXzlh0P/nxjccxXsPGAYChjr6els5nTsPIFU30jQDg2qMoE30jTn3tTLeJn/Ufdj/lsduqGbNHTbWzsHmakSgQCMiMFggEDQS/pLJUtvcEKYFWXI+bQqE0ed2bC+HxN+4+uxAeDwBUqqqDLc3EUJcgBJU1tYOdekwbN3gGfajkjvO9J6/vPXWjpKzK3MRg9ddeq772olAopRXVvwff2n74ilAomjp28HcLp7gNerd53tBA2NOXxZzeSE76kXT51qNlW06UVdYAgI62xsEfF86d9llzz9Lk66L2nY0XdWpvFAqlyeveMO78Ff7Pncux4UN7D/QZPdVn9DQjXQMA2Hz+F9eezuSJf9Lz31x/HL3/r5PFlaUA8M1kv/Eun3kNGxf1LHb2T8s5dVwTfaOD/tsoFMrsnctszLr9tfF4f7veA76dkFdaNI8+y8Whf+TTu/rauoeW/Y+c63M/+fG0LYtqeXXzPb74Y/l2FRWVKm7Not/+T7yJPcppeMjaAyb6RgkZyb9ePX4pNhwAZo30XDF9wZBeA5p7jRpTHPCLpJTaICU7RlpWfp8e1k3exWvgP01508AnBvfvLp5s1HKYkh2guZT8SJw6bm5JQQ/LTzTUNAAgm5VnaWJO/l3JqdbS0EjNyajkVPeytm90jgxOHbeWV29u+J8dmtmsvMxCppWJRd9PesnQGUxJZdVpUrL9YEp2gHZKSYWCKamsFOtc5QghpGgwJRFCSBpMSYQQkgZTEiGEpMGURAghaTAlEUJImtbNBGrXrsgRTuBob0r85ZGEXySl1IqURKitEASxevVqNpudmpqalJQkeReNRsvLy6NSO+icflIQBGFjY8NisSQbfXx8zMzMhg0b5uHhYWZmJq++oY6EW9xIDqhU6t69e5ctWzZ69OhGd504cUIRIhIAqFTq5cuXGzWy2ewJEyZMmzYNI7LrwFoSyROTyZwyZUpKSgp5k06nR0Up1vnKPDw8yGtU6OrqDhky5OLFi5iPXQ3Wkkg+CII4ePCgvb19SkqK+Exrx441fT1uORJ3ydra+s6dO+bm5gwGQ/oqSMlgSiI5YDKZrq6uAQEBABAcHLxr1y4ACAwMVMBT1dnZ2ZGnr79x48aBAwcAwM/Pz9fXl8vlyrtrqKOIEOpAfD6fzBoAoNPpJSUlIpEoODgYADgcjrx71zQOhwMAwcHBIpEoKSmJRqMBAI1GCw8Pl3fXUEfAWhJ1nEYlZFRUlHgfX3BwsI6OjtS15UZHR2fLli3k305OTnl5eYGBgSwWy9PTE4vKLkHeMY26hCZLSLG4uDg+ny+vvrUEn89vVDliUdl14DFu1O6YTOa0adPIeZHBwcG+vr7y7lHbIGd97t+/HwB8fHzOnj2rIHOYUNvClETtiCCIHTt2bNq0CQDodDqDwVC+aTTJycnjx49nsVg0Gu3vv/+W7eI/SJHhfknUXpKTk21sbDZt2kSj0RrthVQm5J5KHx8fFovl7OwcFBSEFyhWMlhLorbXaFP02LFjCntkpg1FREQsXLgQi0rlgymJ2pjkFuiJEycmT54s7x51HC6Xu3jx4pCQEAAIDAzcu3cv7qlUApiSqM10zRLyfVhUKhlMSdQ2unIJ+T7JovLAgQNLly7ForLzwpREHwtLyOZERER4enoCgLOzc1hYmAL+/hK1BB7jRh8lPj7exsZm//795ORqBoOBESk2efLkkpISOp2elJRkb29/8OBBPPzdGWEtiWQkuVGJJaR0DAbDz88PsKjsnLCWRLKIiIhwcHAICQmh0WhxcXFYQkrn6+uLRWXnhSmJWofL5fr6+np6erJYrMDAwLy8PPHZIZEUZmZmUVFR5NmPAgICJk2axGaz5d0p1CK4xY1aAee4fDw2m+3r60ue/1yZftWuxLCWRC3yfgmJESkbMzOzyMhI8Ql9PTw8sKhUcFhLog/DErI9KOupkpQP1pJIGiwh24+dnV1CQgIWlYoPa0nULAaDsXr1aiwh25tkURkeHt7Ff7akgLCWRE1gs9keHh5+fn5YQnYAyaISrxKhgLCWRI3hFGh5wd/CKyasJdE74hISAA4cOJCQkIAR2ZHw0mOKCWtJ9BaWkIoDi0qFgrUkwhJS4bxfVOIvGuUIa8muDktIRYaXHlMEWEt2XVhCKj689JgiwFqyKyII4vDhwwEBAYAlZCeBP3+SI6wluxwmk+nq6kpGZHBwMJaQncLkyZMzMzOxqJQLrCW7EMkSkk6nMxgMpbxAtnKTLCofPnyI/8N1AKwlu4pGJWRUVBRGZGckWVTiCX07BtaSyg9LSKWEkxM6DKakksPTcykxyRP64vVs2w+mpNLCErKLwKKyvWFKKicsIbuURkXl8uXL5d0jpYJHb5QNQRBBQUH29vZJSUl0Or2kpAQjUuk1uvQYntC3bWEtqVTwLAldHJvNJq9nC7gN0XawllQSZAnp7OzMYrF8fHwyMzMxIrsgMzMzvEpEm8NaUhlgCYkawR3TbQhryc4NS0jUJLz0WBvCWrITwxISfZC4qMQvicywluyUsIRELUQWlVu2bMGrRMgMa8nOR/J8B1gdoBbCLQ+ZYS3ZmXC5XF9fX09PTywhUWvhpcdkhrVkp4HnYUVtAovK1sJaUhExGAzJm5IlZGBgYF5eHkYkklmjovL9E/oePHhQXn1TUCKkYOLi4nx8fMQ3w8PDaTQaANBotKSkJDl2DCkZ8sD3+18tAOBwOHLsmKLBWlKxEATh7+/v5eUFWEKidubk5NTkVSKcnZ3XrVsn794pEnnHNPoP8oQFwcHBWEKiDtPoy+bj4wMA2dnZ8u6XosCjNwqEy+Xq6uoCgJub24MHDwAgMDBw7969eGpV1N64XO7ixYtDQkIAoFevXq9fv6bT6VFRUfLul0LAlFQgQUFB+/fvBwBNTU19ff3o6GjcxEYd6fr16/Pnz9fS0iooKACAuLg4d3d3eXdK/jAlFQWTybS3t5dsodPpK1euHDVqlI6Ojrx6hbqI5OTkq1evbtq0SbLR2dk5ISEBN2Xw6I2iWLx4caOWkpKS8+fPr1u3Dmf/onYVHx9/7969V69e0el0yfakpKTQ0FB59UpxYC2pENhs9sWLF42NjT/55BMrKyszMzOsH5G8cLlcNptdUFCQk5NTXl6O14dQtpSkUCjy7kJHULJPDcmL8o2X9hgaSrjH4do1JU+QqVOV7ZuN5EiZxks7DQ3cL4kQQtJgSiKEkDSYkgghJA2mJEIISYMp2cZEIlF+/it59wIhhcPj1ZaU5Mi7F7LAlGwzNTXlV6/+/M03PX/6aZa8+4KQAikoeH38+Mr5862uXNkj777IAlOyzVRXl1pYdGexsuTdEYQUS21tlaamLpdbKe+OyAhTss1YWfVyc5uppqYp744gpFh69hwyZIiXvHshO0zJNkalqsu7CwgpnE49LjAlP6yujtPcXTxeLQAIhUKC4JMtLfzJl0gk4nAqBAJBSx4ToU6kyfHS8p9CCgQEh1Px/g8NCYJPjgg+v6GDf6GrhL9QfN/jx9euXNnd0FCvp2fs67vV1rZvaOj2xMQYAKDRuk+duqJPH7eKiuLg4B+zsp4NGDBu/vxdAMDhVJ47t66srEBNTSMtLX7QoAnz5v1kaGgOAFxu1e3bZ2/ePOzqOmnIEK89e3x4PO727XccHFwlnzc7O+nEiZVcblWPHi4uLhPd3GYCAI9Xe+7c+vz8dC63gslMHjv2qwUL9mpoaLXwMRFqb20+Xt734MGVy5d3UigqAwfShw6d2rv3MABgsbLPn1/P5VYVFr6uq6vx9t7g5bWcbL9162h09MmAgBMsVtbp02vMzT/57bdnmpoddEaYLlFLDhs21csrICsrQU/PxNFxuLa2/vz5u4yNLbOyEsaNm9+njxsAGBlZuLvP0tU1Ij/y2tqa778fbmJitWFD2Jo1oTt23H3y5NqaNe7V1aUAkJR0Oz39YV7ey9LS/Lt3z7u4TKir41RWFjd6XhMTq/LyIk/P5cuXHyUjUiAgNm+e1KvXsM2bI/bsefjVVzsiI//47bd5LX9MhNpbm4+X95mZ2VZXly5denDu3O1kRLLZeVu2TJo9e+OmTeGHDr3q23fk0aMBERGHCIKflHT72bObVVUl//xzvaqK7eDgWl5eWF/fcacT7BIpCQAjRnibm9slJd0WCN5eVHPKlEAAeP78b/EyT55c9/ZeT/4dHPxjWVnBjBnfkzctLXt++eWPRUWZZ878AACffjpj8mR/AMjNTfX3P7xixelLlziDB//nusZlZYXbtnl9880BOn2+uDEq6qRQKPjss9nkzcmT/dXUNOPjLxUWZrTkMRHqGG07XhpJTY09fHjZzp33evYcIm48c2aNm9ssa2tHAFBVVZ06NQgAQkN3UKlq48cvdHP7HAAAKHPn/m/37vhz50qaq1LbQ1dJSQqFMnbsvKqqkoSEm2SLlpYeANy5c47P5wEAn8/LyPjHyWk0+XdMzOlPPumvpvZul/Po0X4AcO9eMJ/fAABqahoA4OIyQVWVCgAaGtqST1dQ8Pp//5u6ZMn+AQPGSbbHxYVWV5ceOLCI/Hfo0FJjY0tjY6vCwowPPiZCHabNx4vYo0dhZ8/+sHHjDTMzW3EjQfAfPforLS1ePDTCw383NrYCAA6nAv4dGuID5RoaWu332t/XJfZLksaNm3/hwtbbt88MHeoFAGFhv4wbNz8m5vSDB3+OGuX76FHY8OHTySXz81/V1lY1mtOjr29qampTWppXXJxtbd0boNm90Q0N9evXj1FRoVpY2De6Kz//lavrxICA482siqdEQ4qirccLAEB29oubNw+7uEzU1zeVXLikJKehoc7DY+GYMXOa7Ix8z4PZVWpJALCwsHN2HvPkyXUOp6KwMFMkEs2Zs11FRfXWrWMAcPfuubFj55FL1tZWA0BFRVGjR9DVNYYW/D+mrq45a9YPpaW5u3fPbnQUmyAa8vPT2+oVIdR+2mO82NsPHD58xtOn4SEhWySXJIgGACgoUNCh0YVSEgDo9AUEwbt//0JY2C/Tpq0yMbF0dZ2UknI3OfmuqqqasXE3crFu3RwAoLiYWVtbI7k6QTTo6ZmSGwLSeXktHz16blJSzJkzayXbTUysXr16UFiYKdnIYmVnZT3/2NeGUFtrj/ESFHTK2rrPhQtbnzy5Lm40NrYEoMTGXmg0++3JkxuKMB+ua6Xkp5/O1NY2iIj4vago09FxOAB4eCwEgD17Zo8bN1+8mLFxt379RhEELy7uoriRIPglJTmjR/upqqq25LmWLTtibz/wr79+vncvRNzo4jJBJBL+/LMvh/P211pCofDs2bXk9wwhhdIe40VLS3fduqtaWnp7984RnxdGV9ewV6+hRUWZR44ECIVCsrGqih0be4FKVWvfF9kCXSslNTS0Ro6cnZub6un59oJHQ4Z4GRrSKBSVwYM9JZdcsmSfurp2aOj26uoysiUm5rSurqGPz9tLcZJzvysqWJJrCQSChoY6co6ChobWDz9c0dTUPXBg4cuXceQCU6euNDAwz8z8Z9myvidPfhccvGnVqiHdu7toa+s195gIyUsbjpe6uhoA4PG4AGBt3Tsg4HhdXfW2bVMqKt7OdZs7dzuFonLr1pEVK1wYjM0nT36/cqUrOesD/h0alZXyGRqqmzdvlssTt5MtW7b4+GyWsoChoXlqatySJfvJ/cEqKirV1aX29gMHDfKQXMzIiNa374hHj/66efNIVVVJTMyZ9PRHGzZcMzKiAcDDh1eDgzeWlxfm5aXxeHU2Nn20tfXT0h6EhGzOykrgcit5vDojI0s+v/7Bg8scTkVsbCifzzM3tzMzs3Z1nfTyZRyLlfXq1cOXL2Pd3Wf5+W2lUChNPmaTLyEkZIuSfWpIXjpmvMTGXrx0aWdpaW5xcTaVqm5t7VhSwoyLC+Vwyh8+/FMkEvbo4WJp2dPWtl9S0p2SkuyUlHtMZtKCBXvJA0dXr/58/fq++nru69dPAMDR8dPmDua009BQwmsofvBqRwUFr62seolvVlWxKRRKo4NuJKFQmJ39gsOpsLDoTqM1PmAtM5FIlJ2dWFXFtrXta2Ly4b2cjUydqmyfGpIXRRsvfD4vI+MpQTQ4OAwmN7BapZ2GhrKNt5Z86p0dpiRqK0o2XtppaHSt/ZIIIdRamJIIISQNpiRCCEmDKYkQQtJgSiKEkDSYkgghJI2yzSmR77lDOoySfWpIXpRvvOBMoA8Tday4uDgajcbn8zv4eeX9NiMl0QHfVT6fT6PR4uLiOuC5RO0zNJQtJTuYv78/i8UKDQ2Vd0cQUlChoaEsFsvf31/eHZGdsm1xd6T4+PgRI0YAAI1Gy8vLo1K70CmNEWoJgiBsbGxYLBYAxMXFubu7y7tHssBaUnbi/x6xnESoSWQhSf7dectJrCVlJC4kSVhOItSIZCFJ6qTlJNaSMpL8j5FGo2E5iVAjZCFJo9HELZ20nMSUlEV8fHxSUlJ4eDh5My8vz8fHZ/Xq1QRByLdjCCkIgiBWr17t4+OTl5dHtoSHhyclJcXHx8u3YzLALW5Z+Pr67tu3z8zMjJxuRr6HDAaDvEvOnUNIAUgOB/EwYbPZQUFB5F2dCKZkq5EFI7kLUjIlAYDL5ero6MixbwgpCMmxIDlMJIdPZ4Ep+VEapSRC6H2dfZjgfkmEEJIGUxIhhKTBlEQIIWkwJRFCSBpMSYQQkgZTEiGEpMGURAghaTAlEUJIGkxJhBCSBlMSIYSkwZRECCFpMCURQkgaTEmEEJIGUxIhhKTBlEQIIWkwJRFCSBpMSYQQkgZTEiGEpMGURAghaTAlEUJIGkxJhBCSBlMSIYSkwZRECCFpMCURQkgaTEmEEJIGUxIhhKTBlEQIIWkwJRFCSBpMSYQQkgZTEiGEpMGURAghaTAlEUJIGkxJhBCSBlMSIYSkwZRECCFpMCURQkgaTEmEEJIGUxIhhKTBlEQIIWkwJVuNIAiCIJq7q4M7g5BikjIWOt0wwZSUxVdffcVmsxs1xsfH//3333LpD0KKJjQ0NCIiolEjl8v19fWVS38+BqZkq1GpVC8vL2dn5/j4eLKFIIigoKBZs2aNHz9evn1DSEF4e3svXLgwKChIXDkmJyc7ODh4eXlRqVT59q21KCKRSN596HwIgrCxsWGxWORNZ2fnpKSk4ODgzvj/JELthMFg+Pn5kaODbKHRaHl5eZiSXQX5DRDf7KQfP0Ltp1ExAQCdtJLALW4ZeXt702g08c29e/diRCIkiUql7oCTdh0AAA3wSURBVN27V3yTRqN5e3vLsT8yw5SUkeQ3oPN+/Ai1K8liovNWEpiSshN/Azrvx49QuxIXE526ksCUlB35DejUHz9C7Y0sJjp1JdFZ+60gvL29DQ0NO+/Hj1B7o1KpJ06c6NST5Drx8KZQKPLugmLB6QqoERwjjcg2RjpxSgLAufif5d0FRTHX/Tt5dwEpIhwjYjKPEdwviRBC0mBKIoSQNJiSCCEkDaYkQghJgynZ0fBINEIfpFDDBFOyo106EtnA48u7FwgpLm51bdiZaHn34h1MyQ5Vzq6KvvJA3r1ASKFdOxvDq22Qdy/eUfKUrOPWy1y4iUQibnWtUCBs8l4BIWjtI/PqG35be7qOW9/kvUKhsL6O19yK5B9YhKI218DjN/edbAnpQ6y+tumvtBSPY15EhNyT4QHJYSIUCglC0Nonla5zzyqXIvrKg4T7yQCQnpjdZ1CPr//vc1OaMTM9P/Li/UJmCQBY21vMWTGdXVQecuB6XS1vkHufGQve/oiKV99w6UhkUS6bW1Obn8UaMcnVN2CquoYaeS8zPf/qqSiRUFRZVtPAa5jt7znQrW/2q/zYyH8yknPUNdV+PLQMANKeZ92/8SQ/u9imB23J+tmlrIqDG88xX+cDwK6VR1UolFlLJvUeYA8A+W9YV078TVGh1HPr87NZY6YN95ozlkpVBYCMZGbM1YdP7iQeCNt49terD/5+/qnHIP/Nfk29YoRa52VCZkTIPZFI9OZlrqGp/twV0/u6OlRXcK6fv/3q+RsA0DPQ9gmYYmRqcGrPnyUFZTY9aH4BU3X0tcnVmxxi5F0EIbh+NubVizdaOpqZKTkD3fv4Lp+irat161Lss/uptdz6id4j3Se6EnwiPPhu4sM0Pl/w1crpPZ3sboXGXjoaCQAPop5lpuZYWJsu+sEbALg1dZeORJSzq9TUqK+Tmc5De3v7TzYw0gOAWk5dXGRCzF8PBwx3HOTe5/dNwbz6hnUHvrV3tG6rN0o5UzL0cEQtp+77XxarqKi8ScvbuvTgjuWHd5xdbdfbeuGaL34KOpKRzBww3FFHT0tLpxunpnbCFyM/8xxCrisgBHtWHR8389M5QdMA4Nal2PO/hdVUcgP+9xUApD3P2r/+zIZDy6zsLIRC4Vq/Pb+uPb3x8PIefW0/6WW5Yub22po68nH6DOph06Pbt5M2qqqqAIApzWjz0cCdgYdfJmSu+XWJZOZuX/7Hd3sW9h7YHQD+uZu0f/1Z5qv8oJ3z2UXl+W9YCbEp/Abi2rnbNj0sjUyzivNLO/79RMrnxYO0a2djvvt5obauVk0Vd8uSA7tXHdt8LNCul5VfwFTGgeuRF+71craz6d4NAAyMdTU11Ret8xb/5LG5IaaprSEUCvf9cNrAWG/tvm8oFMrDqOeHNgdXl3NW7V4w4YuR6upqJ3dfriyvAQCqGnXafHpVRU3U5fhabj0ATPAeadfb+n/+v7t5uHzp70k+Vx23fvOS/SMmuM7/7nMAYOWxtyw5kJ74ZtPRQD1DnZcJmRkpzEJmsU0PWvytZ07DesdFPq0qr2nD90oJt7iLckpuXrzvs3yKiooKAHTvY9N7gD27qPxh1HMAUNdQW7ZljrauZkTIXVZ+6c2L9/sP7imOSAC4F/6PUCj8lD6IvEmf4aamTn1yJ4mVxyb4xJFtISMnDbGyswAAFRWVQe59hQJhXlYReVPPQFuyJ7r62hSVD/yQ9tjOUIf+n5ARCQBDRju7juyXEJv64O/n5pYmY6YN797HBgDselt5zRnzy5/rfzy8vK3eKNRlCQjBqT2XZy4cr62rBQB6BjojJrkKCEEE4y65wJffTu7R1/Z1EjP+1rOsl7m5GUVfr5kljkjpQ+zutccp/7z+0t+TXL6vq4OGpjrzdQG5rp6RbqPO6L/X0sjlYzcr2FWefqPJmzQbs+lfexQXlF08HA4Ag0c50We6A0DBm+Kvv//8mw2zj8fsGOjW52PfIwlKWEs+vpOoSlU99+tf4pY6br2RqX5ZcSV508TCcN7qmX9sYRz88Zy2ruaaX5f8Z/WYFzVV3OM7Q8Uthqb6RAPByistLigrK6506G8rvutLf89xM93MLU1k62pmSk5uRqHXnDGSjW4TXBNiU+/feOI+wQUA1NSpADDIrS8AkJvhCH2kzNTc8pKqezeePIp+QbaUlVQameqLSzBVqurSjT4b5v/K2H/NtJtR4PZ5kt896UPsfvg/FtamegY65F0Gxnq/Xd0g81k3+A1EbMRT6+40qtq7sHIb73J+X9iDW8/mr55JVaOqqasCgNOw3qpUVQDQ0FSX9dmapoQpWZRTom+kS+7OaI7beJfncS8fxbyYs2Ka6n+jpyinxHm4Y5OrR4TcBQBNbU1xi4qKiswRCQBZL3MBQE1dTbLRrpcVABQwi/9twNO6oDZWlFMCAD7LvIzNDZtbhmZj5hsw5dSeP3s625lY/Gcx6UMsP5tl3Z0m2aKrr93kki1RmFNSx61vNEb0DHWMzQ3KS6rYheXdPjFv7zGihFvcBF9Qxqr44OFgqpqqjp7W5aM32UXl/1mdEBTlljT3yADALixrq66SxxYry6olG8m94+Idlwi1OfLqr0W5ja8p34iKqoquvnbC/ZQnd5L+s7rUIUbwBezC8ibvkkF9U2ME/k1etQ4ZJkqYkkZmBgKB8GH0c8lGoUCYcD9FfPPutceGpvrfbvKtr+Ud3soQCoWSq2ck57D+e5CkpLCM+brA2NwAAB7fTmz0jPG3npF/tPZ0fhbWpgCQn1Uk2Uh+g217WrbqoRBqOSNTAwCIi3zaqP2fu+/SMC+r6HHMi22nVmrraZ3afbmy9F1OSR9ixuYG1RWcVy+yJO9Nf/GmlFUBACoyjZHSovJG05UIvkDXQNvYzKBVjyYbJUxJp2G9AeDCwRt5EukTefG++r97K3IzCx/ffvHFkkkDPu0zeuqw10nM8OC74iWdh/YWiUSHNgVz/z1aLRQKL/4RYWFl0n9wLxVVlbRnWYmPXomXj7+ZQB7FBgANLXVefYP4/9jqCg6IQHL2looKBQDEczAHuvXR1NbISMlh5b37X704rxQARkwa3EbvB0KNOQ7srqZOjbuZEH8zQdyYkcwUD5n6Wt6pPX8uXj/blGY0b9UMTnXtsR0XxUtKH2IDhjsCwKUjkeJvfn0dL+pKvCnNCAA0tNQBgFPFFa9YXc4BAMG/C5MHPAX/jhFDU33Hgd0JvuBxzLvqhCAEpawKt/EuKqodkWBKmJLOw3o7DurBqa7dtGjfsR0X/zod9cv/nUx7luk0tBcAcKpr/9jMWLjWm3x/fZdP0TPU+fPYzYxkJrn6xNmf6RvqZr/KW+O3m3Hg+p/Hb21cuM+ut5WWjqahqf74L0YAwG9rT53fFxZ9Jf6PrYzb1x4NGe1ErmvdvRsARDDu1nHrU/55/depKKBAcX4p83VBVUUNAOga6ABAwv2UxEevntxJ1NLR9F0+BQBCfr8hrmdjrj5wHu44+LP+5M0GXgMAVLbpzAbUxenoa5PHDA9vC/n5uxNhp6OP7ww9si1kwhcjAUAkEh3dfmGC90iyUnMb79J/aK+kx+nhjLvk6tKHmNecsdq6mq+TmFuWHLh1KfbamZj1834ZMfHt//pWdhYUCuVxTGJhTkllWXXo4QhyF3zas6z8NywAIA/7vHqRxcpjhx6JEAqEc1ZMV9dQCzsTXfNvtsaG/6OtpzXz3wnO5BipKm+8Vd5WVDdv3txOD93etmzZMnNhExfToFAogz/rX5RTkp9dnJNRmPYsy8hM33/LHA1N9dzMwl/+72RBTrG5lUmPvrYA8Cw25VlsagOP//RecgOP6OPSQ1NLw/nT3umJ2ezC8syUnPTE7KFjnD9fPIHcmu4/uGctpy4zJTczNSfx4SttXa2AbXPJ6RQAYGVv8eRO0vO4l9fP3RaJRPNWzbx2NsbQWE9bV7N7HxsNTXU9Q50nt5Me305UU6d6+o2hUCj2jta6Btr3rj958SCtlFVx4/xtXQOdbzbMVqWqVpZWXzsX8+DWcwDITMnR0de2srdo7t24evLvzvtRonbS3BgBAMdBPYQCUWYKsyiX/fJZZm1N3fJtX5lbmdRx6w9sOJcQm6quoTbQrY+KqgozPT/+ZkJ1BSf1aUY5u6pnfzsNLfXmhhgAaOtq9hvc82VCZkF2cdKj9KyXuZ8vmvipx9updZraGg08/osHadF/xsdHJnjNGaOiSklPzNY30rXqbmFuaaJnoMNMz097nvXkTtKMr8eb0IwMTfR6Odsn3E+5/dej6gpObMTTzNScVbu/NjTRB4Cn95IvH7tVWVpdwCxpqG+wtLPQ0tFs8iXLPEYoCnXujVahUCjSz1ZfUlhWnFdqZGbQ6IhbS4hEotyMwupKrpW9xfv7PipLq/PesAxN9cg5t5Lq63hvXuYZmep3+8QcANJfvOnpbEdOK3u7QC2Px2sgfzYgxqtvYKYXiERCKzuanqFOa3sLAHPdv+u8HyVqJx8cI5zq2pzXBRqa6vaO1qqtn2cmZYgJBcLs9Pz6Ol53R5v3Y6uAWVxdXtO9r62Gpjorv1RdQ63RKCsrrjQw0ZOcfiQUCnMzCrk1dWaWxrJNLJF5jChzSnYpmJLofThGJMk8RpRwvyRCCLUhTEmEEJIGUxIhhKTBlEQIIWkwJRFCSBpMSYQQkqZzzwSSdxcUS+f9KFE7wTHSSJebL4kQQh0At7gRQkgaTEmEEJIGUxIhhKTBlEQIIWkwJRFCSBpMSYQQkub/AcAfR5hjVlrYAAAAAElFTkSuQmCC
　　从资源管理的角度来说

• Master  掌管整个cluster的资源，主要是指cpu core和memory，但Master自身并不拥有这些资源
  
• Worker 计算资源的实际贡献者，须向Master汇报自身拥有多少cpu core和memory, 在master的指示下负责启动executor
  
• Executor 执行真正计算的苦力，由master来决定该进程拥有的core和memory数值
  
• Driver 资源的实际占用者，Driver会提交一到多个job，每个job在拆分成多个task之后，会分发到各个executor真正的执行
  

　　这些内容在standalone cluster模式下的容错性分析中也有所涉及，今天主要讲一下资源在分配之后不同场景下是如何被顺利回收的。

资源上报汇聚过程
　　standalone cluster下最主要的当然是master，master必须先于worker和driver程序正常启动。
　　当master顺利启动完毕，可以开始worker的启动工作，worker在启动的时候需要向master发起注册，在注册消息中带有本worker节点的cpu core和内存。
　　调用顺序如下preStart->registerWithMaster->tryRegisterAllMasters
　　看一看tryRegisterAllMasters的代码

def tryRegisterAllMasters() {
for (masterUrl  {
totalMb = 2*1024
System.out.println("Failed to get total physical memory. Using " + totalMb + " MB")
}
}
// Leave out 1 GB for the operating system, but don't return a negative memory size
math.max(totalMb - 1024, 512)
}

　　如果已经在配置文件中为显示指定了每个worker的core和memory，则使用配置文件中的值，具体配置参数为SPARK_WORKER_CORES和SPARK_WORKER_MEMORY
　　Master在收到RegisterWork消息之后，根据上报的信息为每一个worker创建相应的WorkerInfo.

    case RegisterWorker(id, workerHost, workerPort, cores, memory, workerUiPort, publicAddress) =>
{
logInfo("Registering worker %s:%d with %d cores, %s RAM".format(
workerHost, workerPort, cores, Utils.megabytesToString(memory)))
if (state == RecoveryState.STANDBY) {
// ignore, don't send response
} else if (idToWorker.contains(id)) {
sender ! RegisterWorkerFailed("Duplicate worker ID")
} else {
val worker = new WorkerInfo(id, workerHost, workerPort, cores, memory,
sender, workerUiPort, publicAddress)
if (registerWorker(worker)) {
persistenceEngine.addWorker(worker)
sender ! RegisteredWorker(masterUrl, masterWebUiUrl)
schedule()
} else {
val workerAddress = worker.actor.path.address
logWarning("Worker registration failed. Attempted to re-register worker at same " +
"address: " + workerAddress)
sender ! RegisterWorkerFailed("Attempted to re-register worker at same address: "
+ workerAddress)
}
}

资源分配过程
　　如果在worker注册上来的时候，已经有Driver Application注册上来，那么就需要将原先处于未分配资源状态的driver application启动相应的executor。
　　WorkerInfo在schedule函数中会被使用到，schedule函数处理逻辑概述如下

• 查看目前存活的worker中剩余的内存是否能够满足application每个task的最低需求，如果是则将该worker加入到可分配资源的队列
  
• 根据分发策略，如果是决定将工作平摊到每个worker，则每次在一个worker上占用一个core，直到所有可分配资源耗尽或已经满足driver的需求
  
• 如果分发策略是分发到尽可能少的worker，则一次占用尽worker上的可分配core，直到driver的core需求得到满足
  
• 根据步骤2或3的结果在每个worker上添加相应的executor，处理函数是addExecutor
  

　　为了叙述简单，现仅列出平摊到各个worker的分配处理过程

      for (worker > workers if worker.coresFree > 0 && worker.state == WorkerState.ALIVE) {
for (app  0) {
if (canUse(app, worker)) {
val coresToUse = math.min(worker.coresFree, app.coresLeft)
if (coresToUse > 0) {
val exec = app.addExecutor(worker, coresToUse)
launchExecutor(worker, exec)
app.state = ApplicationState.RUNNING
}
}
}
}

　　launchExecutor主要负责两件事情

• 记录下新添加的executor使用掉的cpu core和内存数目，记录过程发生在worker.addExecutor
  
• 向worker发送LaunchExecutor指令
  

  def launchExecutor(worker: WorkerInfo, exec: ExecutorInfo) {
logInfo("Launching executor " + exec.fullId + " on worker " + worker.id)
worker.addExecutor(exec)
worker.actor ! LaunchExecutor(masterUrl,
exec.application.id, exec.id, exec.application.desc, exec.cores, exec.memory)
exec.application.driver ! ExecutorAdded(
exec.id, worker.id, worker.hostPort, exec.cores, exec.memory)
}

　　worker在收到LaunchExecutor指令后，也会记一笔账，将要使用掉的cpu core和memory从可用资源中减去，然后使用ExecutorRunner来负责生成Executor进程，注意Executor运行于独立的进程。代码如下

case LaunchExecutor(masterUrl, appId, execId, appDesc, cores_, memory_) =>
if (masterUrl != activeMasterUrl) {
logWarning("Invalid Master (" + masterUrl + ") attempted to launch executor.")
} else {
try {
logInfo("Asked to launch executor %s/%d for %s".format(appId, execId, appDesc.name))
val manager = new ExecutorRunner(appId, execId, appDesc, cores_, memory_,
self, workerId, host,
appDesc.sparkHome.map(userSparkHome => new File(userSparkHome)).getOrElse(sparkHome),
workDir, akkaUrl, conf, ExecutorState.RUNNING)
executors(appId + "/" + execId) = manager
manager.start()
coresUsed += cores_
memoryUsed += memory_
masterLock.synchronized {
master ! ExecutorStateChanged(appId, execId, manager.state, None, None)
}
} catch {
case e: Exception => {
logError("Failed to launch executor %s/%d for %s".format(appId, execId, appDesc.name))
if (executors.contains(appId + "/" + execId)) {
executors(appId + "/" + execId).kill()
executors -= appId + "/" + execId
}
masterLock.synchronized {
master ! ExecutorStateChanged(appId, execId, ExecutorState.FAILED, None, None)
}
}
}
}

　　在资源分配过程中需要注意到的是如果有多个Driver Application处于等待状态，资源分配的原则是FIFO，先到先得。

资源回收过程
　　worker中上报的资源最终被driver application中提交的job task所占用，如果application结束(包括正常和异常退出)，application所占用的资源就应该被顺利回收，即将占用的资源重新归入可分配资源行列。
　　现在的问题转换成Master和Executor如何知道Driver Application已经退出了呢？
　　有两种不同的处理方式，一种是先道别后离开，一种是不告而别。现分别阐述。
　　何为先道别后离开，即driver application显式的通知master和executor，任务已经完成了，我要bye了。应用程序显式的调用SparkContext.stop

  def stop() {
postApplicationEnd()
ui.stop()
// Do this only if not stopped already - best case effort.
// prevent NPE if stopped more than once.
val dagSchedulerCopy = dagScheduler
dagScheduler = null
if (dagSchedulerCopy != null) {
metadataCleaner.cancel()
cleaner.foreach(_.stop())
dagSchedulerCopy.stop()
taskScheduler = null
// TODO: Cache.stop()?
env.stop()
SparkEnv.set(null)
ShuffleMapTask.clearCache()
ResultTask.clearCache()
listenerBus.stop()
eventLogger.foreach(_.stop())
logInfo("Successfully stopped SparkContext")
} else {
logInfo("SparkContext already stopped")
}
}

　　显式调用SparkContext.stop的一个主要功能是会去显式的停止Executor，具体下达StopExecutor指令的代码见于CoarseGrainedSchedulerBackend中的stop函数

  override def stop() {
stopExecutors()
try {
if (driverActor != null) {
val future = driverActor.ask(StopDriver)(timeout)
Await.ready(future, timeout)
}
} catch {
case e: Exception =>
throw new SparkException("Error stopping standalone scheduler's driver actor", e)
}
}

　　那么Master又是如何知道Driver Application退出的呢？这要归功于Akka的通讯机制了，当相互通讯的任意一方异常退出，另一方都会收到DisassociatedEvent, Master也就是在这个消息处理中移除已经停止的Driver Application。

    case DisassociatedEvent(_, address, _) => {
// The disconnected client could've been either a worker or an app; remove whichever it was
logInfo(s"$address got disassociated, removing it.")
addressToWorker.get(address).foreach(removeWorker)
addressToApp.get(address).foreach(finishApplication)
if (state == RecoveryState.RECOVERING && canCompleteRecovery) { completeRecovery() }
}

　　不告而别的方式下Executor是如何知道自己所服务的application已经顺利完成使命了呢？道理和master的一样，还是通过DisassociatedEvent来感知。详见CoarseGrainedExecutorBackend中的receive函数

  case x: DisassociatedEvent =>
logError(s"Driver $x disassociated! Shutting down.")
System.exit(1)

异常情况下的资源回收
　　由于Master和Worker之间的心跳机制，如果worker异常退出， Master会由心跳机制感知到其消亡，进而将其上报的资源移除。
　　Executor异常退出时，Worker中的监控线程ExecutorRunner会立即感知，进而上报给Master，Master会回收资源，并重新要求worker启动executor。