Phương pháp giải:

- Tính diện tích tứ giác $CDNM$: ${S_{CDNM}} = {S_{ABCD}} - {S_{\Delta AMN}} - {S_{\Delta BCM}}$.

- Tính thể tích khối chóp $V = \dfrac{1}{3}SH.{S_{CDNM}}$.

Lời giải chi tiết:

Ta có:

$\begin{array}{l}{S_{\Delta AMN}} = \dfrac{1}{2}AM.AN = \dfrac{1}{2}.\dfrac{a}{2}.\dfrac{a}{2} = \dfrac{{{a^2}}}{8}\\{S_{\Delta BCM}} = \dfrac{1}{2}BC.BM = \dfrac{1}{2}.a.\dfrac{a}{2} = \dfrac{{{a^2}}}{4}\\{S_{ABCD}} = A{B^2} = {a^2}\\ \Rightarrow {S_{CDNM}} = {S_{ABCD}} - {S_{\Delta AMN}} - {S_{\Delta BCM}}\\\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, = {a^2} - \dfrac{{{a^2}}}{8} - \dfrac{{{a^2}}}{4} = \dfrac{{5{a^2}}}{8}\end{array}$

Vậy ${V_{S.CDNM}} = \dfrac{1}{3}SH.{S_{CDNM}} = \dfrac{1}{3}.a\sqrt 3 .\dfrac{{5{a^2}}}{8} = \dfrac{{5{a^3}\sqrt 3 }}{{24}}$.

Chọn B.