The XIAP protein is a member of apoptosis proteins family. The XIAP protein plays a central role in the inhibition of apoptosis and consists of three Baculoviral IAP Repeat domains. The BIR3 domain binds directly to the N-terminal of caspase-9 and therefore it inhibits apoptosis. N-terminal tetrapeptide region of SMAC protein can bind to BIR3, inhibit it and subsequently induce apoptosis. In this study, fifteen tetrapeptides were docked into the BIR3 domain and then 10 ns molecular dynamics simulations were performed on each of the BIR3-peptide complex obtained from docking. MM/PBSA method was subsequently used to calculate the binding free energy of peptides to BIR3. The results of MM/PBSA method were in good coordination with docking and existing expermental results.  The results showed the most potent peptides with the lowest binding free energy for binding to BIR3 included ATPF, AKPW and ARPF peptides. Also, investigation of bonds between these peptides and BIR3 domain in the final structure of complexes showed that Leu 307, Thr 308, Glu 314 and Tyr 324 of the BIR3 domain were essential for binding of peptides. Energy decomposition results for binding these peptides to the BIR3 domain during MD simulation was inconsistent with previous results and approved the roles of the same residues. The higher affinity of these peptides relative to native peptide (AVPI) and comparing them with other peptides revealed that the existence of positive charge in the second position and the existence of the aromatic group in the fourth position led to more binding affinity.