A search for Higgs boson pair production in the bb¯WW* decay mode is performed in the bb¯l?qq final state using 36.1 fb-1 of proton-proton collision data at a centre-of-mass energy of 13 TeV recorded with the ATLAS detector at the Large Hadron Collider. No evidence of events beyond the background expectation is found. Upper limits on the non-resonant pp ? HH production cross section of 10 pb and on the resonant production cross section as a function of the HH invariant mass are obtained. Resonant production limits are set for scalar and spin-2 graviton hypotheses in the mass range 500 to 3000 GeV.